DEV Community: Hannah Aubry

Fastly and the Linux kernel

Hannah Aubry — Mon, 24 Jun 2024 15:16:00 +0000

Today, Fastly announced a $40M commitment of free CDN, compute, and security services to the Linux Foundation’s ecosystem of maintainers, furthering our shared mission of sustaining free and open source software. To celebrate, we wanted to highlight an essential component of the Fastly platform and part of the Linux Foundation’s original raison d’être: the Linux kernel.

When we launched Fastly 13 years ago, our founders were fueled by a vision to create a radically different kind of platform—faster and more efficient than its predecessors, utilizing specific hardware and a foundation built on open source software. The tools they reached for? SSDs, Varnish, and the Linux kernel.

Our founders opted for open source, like the Linux kernel, whenever possible for the same reasons anyone does. We could quickly fix bugs without needing to call a support line or wait on a ticket. We could change the kernel’s code so that it was even better suited to our needs. Perhaps one of the most important reasons is that we could not only stand on the shoulders of giants but contribute alongside them, too. Throughout the Linux Foundation’s almost 25-year history, the organization has provided vital support and resources to the maintainers and contributors that build some of the most important and impactful technologies of our time. As of 2023, the Linux Foundation is home to over 1,000 active projects and foundations ranging in focus from cloud computing and web development to visual effects and IoT devices.

The open source projects Fastly uses and the foundations we partner with are vital to Fastly’s mission and success. Here's an unscientific list of projects and organizations supported by the Linux Foundation that we use and love: The Linux Kernel, Kubernetes, containerd, eBPF, Falco, OpenAPI Initiative, ESLint, Express, Fastify, Lodash, Mocha, Node.js, Prometheus, Jenkins, OpenTelemetry, Envoy, etcd, Helm, osquery, Harbor, sigstore, cert-manager, Cilium, Fluentd, Keycloak, Open Policy Agent, Coalition for Content Provenance and Authority (C2PA), Flux, gRPC, Strimzi, Thanos, Linkerd, Let’s Encrypt, WebAssembly. And the list goes on!

Because the Foundation is a fundamental component of Fastly’s tech stack, we are delighted to give back to the community that maintains and supports it. We do that in several ways, the chief of which is our donation of free services to various projects under the Foundation. Since 2015, Fastly has served nearly 35,000 terabytes to Linux Foundation projects, including the Linux Kernel, Kubernetes, jQuery, and Jenkins. We’ve heard that the projects we support have done wonders for their peace of mind and the continued sustainability of their projects, so we’re recommitting to our existing partners and extending our commitment to any other projects or nonprofits that can benefit from our support.

We contribute back to the Linux ecosystem in other ways, too. Beyond our in-kind donation, we encourage our employees to submit bug fixes upstream and employ several engineers who contribute to various projects, including Linux kernel contributor Joe Damato. His contributions are focused on improving the granularity and ease of customizing and optimizing the data path, among other things:

Recently, Joe upstreamed a major and widely anticipated new interface to the kernel, allowing programs to control network socket busy polling behavior more simply and closely. This new interface allows Fastly and the open source community to build and control latency-sensitive programs far more easily.
Joe recently upstreamed a change to the Mellanox mlx5 driver, which creates a mapping between hardware network queues on the device, their associated interrupt number, and the kernel’s internal NAPI ID. This change allows users to access this helpful information via a new netlink interface added to the kernel by Intel.
Another recent highlight among Joe’s many contributions is a kernel change that allows users to set a custom flow hash on custom RSS contexts if the driver supports it. This change also allows network device driver maintainers to support this for their devices, providing users more flexibility in how network flows are mapped to device queues.

And as far as Fastly’s kernel goes, it’s laser-focused on instant speed. Suresh Bhogavilli is the distinguished engineer who leads our Edge Host Networking team, which is tasked with optimizing our kernel and reducing latency by every zeptosecond possible:

To significantly reduce round-trip time and improve customer connectivity to Fastly’s edge we developed our Precision Path and Fast Path Failover technologies. We use them to tune TCP window parameters per destination, improving TCP connectivity between caches at our own PoPs [Points of Presence]. Linux also helps power our ingress traffic engineering tool, called Harmonizer, which allows us to carry traffic to the PoP that is most optimal to serve that customer request.

No more montages

Hannah Aubry — Sun, 28 Jan 2024 18:43:05 +0000

I’ve been a hypocrite.

For years, I’ve worked hard to advocate for the open source community, to encourage others to live and work by its values: openness, transparency, and collaboration. Meanwhile, I haven’t lived those values fully. The majority of what I’ve written, here and here, is about finished projects, or celebrating some milestone. I haven’t shown my work. I haven’t shared how I think about my work or written about any of the messy stuff or my failures (which is where I’ve learned the most).

So I made this place to change that — I’ll do my work in public and write about my own philosophical noodlings here. I hope someone out there finds it useful. Maybe you, friend.

This is the work

Since Fastly launched Fast Forward (that’s our open source support program) one year ago, I’ve paid a lot of attention to those two words wherever they appear. (You can probably imagine that’s pretty often for me.) Usually, I smile to see them, but there’s one place where I’ve come to dislike seeing “fast forward.” In stories. I’m bored out of my brain of that storytelling mechanic where they lay out a nice, juicy problem… and then skip past the glorious work.

No 👏 more 👏 montages. 👏 (Sorry, Team America.)

I joined Fastly to work on their open source program almost four years ago — my start date was March 9th, 2020. I love to build things with passionate people, and I love to help others build too. I’m hard-pressed to think of a community that is more passionate and committed to building and doing than those drawn to working in open source communities. Within a month, I worked on the announcement of Fastly’s $50 million commitment to the program, and the expansion of its scope to include nonprofits, to support those fighting COVID and its effects on society.

It’s been a long, difficult, and glorious road since then.

The $50 million commitment to Fast Forward opened a floodgate of applications that effectively stress-tested the program’s internal infrastructure & resourcing. Meanwhile, managing the program was just a part of what I was doing at Fastly. I won’t get too deep into the nitty-gritty, but the basic gist of what was broken was that our processes weren’t built to scale, the program’s work wasn’t tied to Fastly’s overarching vision of what the internet can be, we weren’t active in the communities we support, and we weren’t taking credit for our work.

So I created a slide deck outlining these issues and proposing fixes to them. I shared the deck with anyone who would listen. I presented it to Fastly’s leadership. It took a long time, and I began to despair that anything would change. Finally, Fastly acquired Glitch. I was moved to the newly formed Developer Experience team to focus on the program full-time. I gained the support of an industry-leading team that deeply understands this work, better than almost anyone — I mean just look at what they built. I began to make inroads with the teams that could help fix the broken stuff. I made new decks. I wrote design docs. I invited comments from anyone who had ideas and verve. There were missed opportunities, false starts, missteps, dead ends. Nothing about it was easy. It was fun, but it was messy, hard work. A full year passed between the time I made that first deck and the day we launched the new program, the program we built together: Fast Forward.

Here’s what I learned:

If you’re trying to create systemic change in any kind of organization, you have to be ready to repeat yourself. A lot. I think most people don’t understand that about communications; I’ve seen this often since my journey in tech began. Your message isn’t sinking in until you’re sick of repeating it.
None of us can do this alone. I think about this proverb often: “If you want to go fast go alone, if you want to go far go together.” Building for a community means building in community, so ask for help when you need it. Trust that if you put your need out in the world, help will come to you. It may not be exactly when you think you need it most, or be in exactly the form you thought you needed, but help will come.
Perhaps the most significant shift I’ve had in the way I think about my work is to stop thinking about it as a series of goalposts. Rather, I now take joy and a sense of accomplishment in the day-to-day moving forward. I see each little step as a win. I see the act of taking a step as a win.

A quick interlude, lamentoso

We’ve had so many hard years: disease, layoffs, wars, relicensing, paywalls, shuttered programs, and 60-day warnings. Climate crisis. The digital spaces where we used to convene and commiserate have changed forever. Institutional Tech lost a lot of ground in building trust with not only the open source community, but the world. It will take time and effort to rebuild and repair. Some people and things are gone forever. We must give space to mourn who and what we’ve lost.

Twenty-six months ago, I went through a period of significant personal loss. Every year since, the holidays have felt dimmer, and I haven’t felt much like celebrating, so instead I take that time to refocus myself. I reflect on the year that passed, record what I learned, and set goals for the coming year.

The most impactful thing I’ve done as part of my holiday reflections is to write down a theme for the coming year. It’s deeply personal to me so I’ve never considered sharing it before — but this year I will.

This is the work. Onward, we must climb. I act when I’m at my best, otherwise I rest. The pounding and lapping of waves on a beach.

(Be gentle with me, I feel vulnerable and exposed.)

History cycles. Time is a wheel. We’ve fought the fights we’re facing before, albeit wearing different mustaches. In the end, we move forward. It’s the best and only way to go.

I’m holding space for anyone who needs a friend. I need one too.

Work in public

Just last week I broke Fastly’s Mastodon instance, Fastly.Social, while updating it to v4.2.4. I created the instance as a place for experiments, a place where Fastly employees could learn together, to demonstrate best practices of using Fastly in front of Mastodon. However, I am decidedly not an expert at this, so to be honest, it was only a matter of time before I broke it. Breaking the instance sucked. It was embarrassing to admit defeat when I couldn’t troubleshoot the issue, especially to admit it in front of my colleagues who quite literally build the internet. But that wasn’t my failure. My failure was that I didn’t apply the learnings I described above until last week.

When I realized I wouldn’t be able to fix the issue, I put a call for help in our interest-fediverse Slack channel. The response was overwhelming; people are wonderful.

I realize now that my superpower is this — it’s not that I’m really good at some things (although, at the risk of sounding like a braggart, there is some stuff I’m really good at). It’s that I unashamedly, sometimes even gleefully, narrate what I’m bad at and I’m always willing to point at stuff that’s broken to those with the power to fix it, even if I’m part of why it’s broken.

People can’t fix what they can’t see, and no person on this planet is omniscient. If we want to fix this broken and beautiful world, we have to be willing to talk about the work in progress. The messy stuff. Recently, I saw someone post, “It’s worth it to be hurt sometimes, so you don’t have to live with your guard up all the time.” Be vulnerable.

And with that in mind — let’s hear what it took. What did you give, what did it cost? Tell us what you need. Work in public. Share what you learned. We can’t wait to celebrate until we reach the finish line, because, for us, there isn't one. The goal is to do the work, to strive for the cause, and to keep making the world a little more open and free, forever. I think doing that is a thing worth celebrating, any and every day.

There is so much work to be done, and the first and most important step is easy. It’s just to care. Be kind, be curious. Think twice. When you see something broken, fix it. If you see someone doing ridiculous amounts of stuff without complaining, help them. You have experience and passion. We need you.

I’ll end this transmission with a little reminder, which is just as much for me as it is for anyone reading — we have to heal ourselves before we can heal the world. Good luck out there and take it easy, friends 👋

Fast Forward Rewind: it was the first year

Hannah Aubry — Thu, 21 Dec 2023 21:46:57 +0000

To sum it up — wow. It’s been a fantastic first year for Fast Forward (say that five times, fast). We’ve reached 100+ program members, and since we launched Fast Forward last year, we’ve served over 5 trillion requests for open source.

But for us, numbers like that aren’t the only way we measure our success. We also measure success by impact, because Fast Forward is designed to help open-source projects and nonprofits during their toughest moments and their biggest successes. Moments when maintainers question whether they can go on or keep up. Moments when they wonder if anyone sees and appreciates their toil or the sleepless nights they’ve given. If we can make a maintainer feel supported and celebrated during a moment like that, then we've won.

So this one is for the builders (we always are). They worked hard. They didn’t get into open source to be people managers, coaches, fundraisers, or communicators. They may not have had any goals at all except to revel in the joy of creation. They worked weekends reviewing PRs and grew gray hairs fending off DDoS attacks. They made us happy and made the entire internet better, or easier, or safer. They deserve our gratitude for everything they do, from the public repos to the invisible labor to the private sacrifice.

So here are a few thank you notes, from all of us at Fastly.

To the social web

You began 2023 in a fever dream. Social media as we knew it was crumbling around us. You were inundated. You were targeted and attacked, technically and sometimes socially. But you stepped up, buckled down, and did the work — you protected us, and you connected us all. Thank you.

To OpenTofu

When this year started you didn’t exist. You were born of necessity — responding to the community prompted your creation, but you rose to the occasion. You rolled up your sleeves, sent out the call, and put your fingers to your keyboard. You built a community overnight, and you’re just getting started. And hi, OpenBao, it’s nice to meet you. Thank you.

To Rocky Linux (and all freeloaders)

Goodness, people made Decisions™ this year. It was just a few months ago that your entire future was called into question. But you rallied the community, and adapted to the changes with grace. Thank you.

To Kubernetes

Few projects understand what it means to operate and manage an open source project at scale the way Kubernetes does — with everything that entails. Keeping internet services up is a difficult, sometimes lonely task. But that daunting task didn’t stop you from striving every day to ensure that everyone all over the world can rely on you. Thank you.

To jquery

Change, especially unexpected change, is hard. You faced that this year, making a seamless transition while handling enormous scale and importance to the web. And you made it look easy. Thank you.

This list is far from comprehensive, but it’s a start. If you’re reading this, I hope you reach out to the maintainer of an open source project you use directly and say thank you. Or, even better yet say thanks and how can I help?

What’s next for Fastly

We’ve got big things cooking in support of open source for the coming year, and I can’t wait to start rolling them out. We’ll keep helping scale and secure the open web, of course, and we’re already exploring new ways to use our platform and resources to support and celebrate maintainers. If you have an immediate need that Fastly can help with, let us know. Even if it’s not bandwidth!

In the meantime, come join our community. We’re hanging out on our forum Fastly Connect, hosting livestreams on YouTube, and attending events. Next on the calendar for us is KubeCon/ CloudNativeCon in Paris, and PyCon in Pittsburgh. Hope to see you there. :)

Now let’s go build the good internet — together ⏩

Surprise! Here's a gift from Kubernetes and Fastly

Hannah Aubry — Wed, 01 Nov 2023 17:18:18 +0000

Your gift:

the fastest and safest Kubernetes (k8s) download experience you can possibly imagine.
warm fuzzy feelings, because you’ll be saving the k8s infra Special Interest Group (SIG) headaches and cash money.

And all we need is a trivial amount of your time to do update any hardcoded references of the Kubernetes soon-to-be-retired GCS bucket’s hostname to their new download endpoint:

That's cdn.dl.k8s.io, for the Ctrl+C, Ctrl+V crowd. It's me, I'm crowd.

Let me explain.

I lead Fast Forward, how Fastly helps support and empower everyone to build the open internet together, and we proudly accepted Kubernetes to our Fast Forward program earlier this year. It was an absolute no-brainer for us. Kubernetes is an outstanding example of open source at its best, with lived values that both our organizations share like inclusivity, trustworthiness, openness, and leadership. And of course, like most globally available and distributed software deployments running at massive scale, we use Kubernetes throughout our stack.

After we accepted Kubernetes to the program, the K8s Infrastructure SIG began the process of deploying Fastly in front of the GCS bucket hosting their binaries. They notified their userbase to prepare for the change by adding Fastly’s IP space to their allowlists, and got started working on their Fastly configurations. Then came the time to instantaneously send over 5 PETABYTES of monthly traffic, roughly 115 gigabytes a minute, to Fastly’s network. So Arnaud Meukam (a chair of the Infra SIG) and Ben Elder (a technical lead) convened with Fastly’s Mission Control team to get it done. Our Mission Control team is like Air Traffic Control for Fastly’s network. They monitor the network during huge live digital events like the Super Bowl, merch drops, or ticket sales to make sure nothing goes sideways.

Once we were gathered, Arnaud deployed the DNS changes necessary to begin routing traffic through Fastly’s caches. Everything was going very smoothly — traffic graphs were going up, binaries were being cached — but then Arnaud and Ben noticed something:

Watch 👀

Growing up in an unplanned way.

By way of a brief history, the Kubernetes project was started at Google around 10 years ago. Google engineers started building Kubernetes in 2013, and it was first released in June 2014. In 2015, Google and the Linux Foundation worked together to create the Cloud Native Computing Foundation (CNCF), with Kubernetes becoming its seminal project in 2016.

In the years since Kubernetes usage has exploded. According to a report from the CNCF, Kubernetes “is the second largest open source project in the world after Linux and is the primary container orchestration tool for 71% of Fortune 100 companies.” There are more than 74,680 devs who contribute to Kubernetes, and 5.6 million developers who use it (that’s 31% of all backend developers).

But as the community around the project grew, the way the project hosted and served its binaries to their global user base remained the same. The binaries stayed in a Google-owned bucket, to which few in the community had any visibility, much less access. Now, Kubernetes is planning to migrate to a new, community-owned bucket. Deploying Fastly is the first step in accomplishing that. Once as much traffic as possible is captured, they’ll spin down the old Google bucket and migrate over to the new one. With Fastly in front that transition will be invisible, as long as you’re using the abstracted domain. So update your references, or your stuff just might break!

How we can help the internet’s plumbers.

If you’ve spent any time working on or near the internet’s plumbing — maybe you’ve worked on building the pipes, or maybe you direct stuff through the pipes — you know how inscrutable and difficult this work can be. Especially when you’ve reached the size of Kubernetes, and even for scale and backend experts like the Infrastructure SIG working on Kubernetes. So any hindrance to accessibility or visibility? Scary.

When I asked Arnaud how he and the team felt managing this crucial piece of infrastructure, on which so much of the internet relies, he (graciously) said, “Serving 5 petabytes is a measure of success, regardless of how we want to present it. We’re struggling through that success. We want to make sure we guarantee the sustainability of our infrastructure over time, and it’s really difficult to maintain it as an open source project. We don’t pay people to do that.”

Elsewhere, he joked:

So I guess you could say it’s a mixed bag.

The immediate and majorly impactful thing you can do to help the K8s Infrastructure SIG is update your CI/CD pipelines, project build workflows, or any OSS repos you have that reference Kubernetes’ GCS bucket hostname to cdn.dl.k8s.io. If you don’t have a project that references the bucket’s hostname, maybe you could submit some PRs to a few of your favorite projects!

For Arnaud and the team, “having access and the capability to use a CDN, to be able to distribute binaries, without needing to worry about what’s happening to the network… it’s a good thing.” It occurs to me that while Fastly’s gift of bandwidth to the project is a big upgrade to their downloads infrastructure in terms of speed, security, and observability, perhaps the best gift of all is merely abstracting away the workload. It's the fact that there's one less thing to worry about, and when things go wrong, there are a lot more people worrying about it along with them. Because let's be real, it's a very fine line between stuff just working and chaos.

(Or less dramatically, a lot of stress and sleepless nights.)

So another thing we can do to support the internet plumbers is roll up our sleeves and help. Sometimes that may be code, or sometimes that may be care — in the form of a thank you, or coffee, or ko-fi. Whatever form your contribution takes, it is meaningful and important. And besides, isn’t this exactly what makes free and open source software, and this community, so special? It’s all of us, from all over the world, building together and helping each other. I think it’s a pretty unique thing, and it's a very special way to be in this world.

As the Infrastructure SIG’s community outreach to get references switched over to the new cdn.dl.k8s.io endpoint continues, you can watch their traffic tick up in real-time on our Developer Hub thanks to our Edge Observer dashboard. Binaries go brr!!

—

Oh! And if you know (or are!) a maintainer who needs help with scale, send them our way. We're here to help.

Now let’s go build the good internet — together ⏩

Community Spotlight: Casey Kolderup, creator of Postmarks, has joined the W3C’s Social Web Community Group!

Hannah Aubry — Mon, 25 Sep 2023 19:18:27 +0000

If you regularly read my stuff, you know I have a lot to say about how we can save the web, and what it will take to do it. We’re blursed to live in interesting times — in the midst of not only a social media revolution, but a revolution of all mass media. One powerful tool we have to affect change in media is the technological systems that underpin it. Building systems that are fundamentally safe, that protect people, especially marginalized communities, is imperative to building a better web. One of those technological systems is ActivityPub, the most widely used protocol in the burgeoning social web — the Fediverse.

The W3C ActivityPub Recommendation was finalized on January 23, 2018, but due to re-invigorated interest in decentralized social media platforms that are not owned by corporations, the Social Web Community Group (CG) has started meeting again. They plan to address issues and advance solutions facing the entire community, most recently at the W3C’s annual gathering, TPAC. A fellow Fastly employee and all-around awesome human, Casey Kolderup, joined the Community Group’s meeting — I wanted to learn more about the community group, so I cornered Casey in the office kitchen* and asked them a few questions.

[*...No cornering was needed to get Casey to answer my questions, they did so very graciously, and we work on opposite sides of the US, so there’s no kitchen... Anyway!]

Hannah: When did you first get involved with the social web? What excites you about it?

Casey: I've been finding communities online since I was joining newsgroups and IRC when I was a pre-teen, but for this modern fediverse-enabled social web I've been instance-hopping around to various degrees since 2017, sometimes public, sometimes private. The fediverse is newly energizing to me because there are a few big platforms that can draw people in with broad utility like Mastodon and Pixelfed, but they can interoperate with things that are similar (comparing something like Firefish to Mastodon, for example) or that are fairly different (like Bookwyrm). Each community can decide on its own standards and also what they want to get out of the software they're using, which is really cool.

Hannah: It’s wonderful, isn’t it? I think it reflects how humans form communities in person more than any social software I’ve seen before — well, maybe forums. So why did you want to join the social web community group?

Casey: I think it's great that an organization like the W3C recognizes the lasting importance of the concept of a "social web" to the internet at large, and having a standards organization that has been working in tandem with both companies and also the indie web community seems like a great alternative to the turmoil we've experienced in the last few years over closed corporate platforms.

Hannah: Completely. What do you want to accomplish by joining, and what are some of the biggest challenges the social web faces?

Casey: For now I mostly want to learn more about what the folks who have already been doing the work are thinking about and try to understand where things are headed. I've spent some time in the past 12 months reading the ActivityPub spec and its associated documents & protocols, but it was with the perspective of an end goal in mind (making Postmarks work and getting basic interoperation with Mastodon in place). Understanding the larger and more forward-looking perspective seems important to me right now!

As for challenges the social web faces, based on what I do have vision of right now, it's all stuff that I hope is, or will become, familiar to people who have worked more in the open source world than I have: How do we bring in a wide variety of backgrounds and perspectives, while prioritizing the safety of people who have been historically excluded & hurt? How do we take something that is so powerful and open-ended and enable developers to refocus it as packaged software that is useful to people & a joy to use?

Hannah: It’s a daunting task, but with thoughtful and brilliant people like you working on these issues, I believe we can overcome those challenges! I want to ask you about Postmarks but before I do — can you tell us what happened during the Social Web Community Group meeting at TPAC?

Casey: While I unfortunately didn’t get to attend in person, the vast majority of the participants did, which seemed like a good opportunity for them to move forward on some discussions that have been in progress about a few upcoming proposals, mostly very specific details of the ActivityPub spec that some people feel would improve its clarity or usefulness–but as with any controlled standard, figuring out how best to revise is difficult. Can you make the revision non-breaking to existing implementations? If so, will that diminish the usefulness of the change?

The other topic that came up was somewhat related: the idea to request a separate “working group”, which is a W3C designation that has different membership requirements and powers, was floated. The proposal as it has been made is structured around incorporating the errata mentioned above (and what sounds like some previous material of a similar scope) into the ActivityPub and ActivityStream specs; from what I understand it will need to be discussed more formally at a future meeting. Since the meeting, there’s been lively discussion about it among individual members of the group and the community at large on Mastodon and a thread for comments has been opened on the ActivityPub SocialHub forum.

Hannah: Interesting — sounds like the beginning of a long and winding road. like Now Postmarks! I know you just released it a week or so ago. Tell me more about it— what does it do, and are there any hidden features people should know about?

Casey: Postmarks is a social bookmarking site that runs as a single-user website. You can use it like a Pinboard account or a linkblog, but you can also interact with the fediverse while you're doing so, sharing your content out in real-time to Mastodon users or watching for cool links from any ActivityPub platform that supports sharing URLs. It's still in its early days, but I've been experimenting with different versions of it for almost a year now and I'm now hoping to get more people involved so we can try to make it useful, fun, easy, and safe for as many people as possible.

It's not a super "hidden" feature necessarily, but we don't really telegraph it on the site right now: every Postmarks site also publishes its own Atom feed! RSS and its related formats were things that really got me excited about the web around the time I moved from working on legacy/proprietary software to working on the open web. You can even run a Postmarks site without ever enabling its ActivityPub functionality but still make it easy to keep up with via its feed.

Hannah: That is so cool! I can’t wait to play with it. [Readers — you can learn more about Postmarks here and find the repo here.] Last question — where should people go if they want to get involved in the Social Web CG too? What recommendations do you have for people who want to contribute?

Casey: Lately I've been lurking on the ActivityPub.Rocks SocialHub, which I did not know existed and wish I had during all this time that I was working on Postmarks! There's more general info on the ActivityPub.Rocks site, and I cannot recommend enough Darius Kazemi's guide to learning about ActivityPub – it was this guide that helped me get from zero to working demo, and I'm slowly picking out all the code from Postmarks that I copied over from Darius's demo server that he has graciously made available.

—

I’m excited to follow along with Casey as they help advance this important and impactful corner of the new web. If you want to follow along with them too, you can find them on their blog, personal home page, or on Mastodon!

Now let’s go build the good internet — together ⏩

Community Spotlight: Keeping open source free with Dirk Müller, openSUSE

Hannah Aubry — Mon, 18 Sep 2023 13:06:00 +0000

There’s an undeniable shift happening in the open source community right now. A quick news search of open source -AI returns headlines like, “licensing firestorm”, “open source vulnerability”, and “battle for the future”. But the outlook isn’t all doom and gloom — especially when you talk to the people hacking away every day, for decades, to further the cause of free and open source software, and making the experience of using open source software more accessible for everyone. In fact, some of them might even say that the future looks bright!

Recently, I had a chance to chat with just one of those people — introducing Dirk Müller, who works in SUSE’s Linux group as Distinguished Engineer, and is a major contributor to openSUSE Tumbleweed and other openSUSE or open source projects. There’s a lot of exciting things going on in the openSUSE community — they’ve augmented their mirror downloads infrastructure with Fastly, making downloads up to 20% faster in areas without reliable mirrors, and just a few weeks ago openSUSE announced they would fork Red Hat Enterprise Linux (RHEL) after Red Hat put access to the binaries for it behind a paywall.

I wanted to learn more about these two developments in the openSUSE community, and learn more about Dirk too, so that’s enough preamble. Here’s our chat!

Hannah: Thanks for chatting with me, Dirk! How did you get involved with open source and openSUSE?

Dirk: I don't quite remember all of it, but I am pretty sure I got involved before the term "open source" was created in February 1998. Some time in the early 1990s I started using DKBTrace (which got superseded by POV-Ray on my Amiga computer. The sources of POV-Ray were available and I started reading and modifying it for myself. I eagerly ported my changes over to newer versions of POV-Ray when they were released and followed the main developers closely. Back then it wasn't a thought of mine to start contributing any of my changes back.

This changed with GoldED, which was a messaging application for FidoNet. This was a Shareware application for MS-DOS that I enjoyed using, but it had a few bugs that annoyed me. As the source code wasn't available, I started modifying the actual executable by applying patches on the machine code level and eventually started mailing Odinn, the main author, my modifications. After a while, he started responding and asked me whether I wouldn't feel more comfortable with modifying the source code directly. So eventually he mailed me an archive of the complete sources and I contributed a few patches. He also mailed me a copy of the GNU General Public License (GPL) v2.0 and asked me to read it and think about helping with a Linux port of GoldED in order to release it in source form under GPL v2. Reading that license I got interested. I was unaware of Free Software and Linux back then; and it took me a while to find an early copy of Linux on a set of physical CDs in a local library (coincidentally those CDs, that I still own, were made by SUSE). I didn't have access to the Internet nor was there good documentation as easily searchable as it is today, so it took me forever to make progress on that effort. Also, Odinn and I made the mistake of developing it behind closed doors and only releasing it as open source when it was "ready", which took far too long, and only happened when we lost interest in the entire project. However, thanks to Odinn’s brave decision to open source it, others took over and the software is still in active use today (in a fork called GoldED+). Seeing that the project survived even though all of the original authors stopped, this "infected" and intrigued me with the principles of open source.

When I started studying at university, I got in contact with a few folks that were starting to work on a free desktop environment called KDE and I became one of the very early contributors to it, for many years to follow. As a community contributor, I also sometimes helped SUSE to solve issues in KDE, as well as reported issues in the SUSE distribution, so I was eventually invited to join an "early beta tester" community of upcoming SUSE releases. Around a decade later, I joined SUSE shortly before the openSUSE project was announced. So in a way I have beencontributing to openSUSE from the very beginning.

Hannah: Incredible, Linux on a CD! Okay, so let’s address it — SUSE announced they would fork RHEL. How has this affected your work? Where do you think Enterprise Linux goes from here?

It does not affect the openSUSE project in any way, also specifically not in regards to the download infrastructure or CDN rollout. There is no change in the openSUSE community mission or near term plans. We recently published a FAQ.

I would hope that Enterprise Linux continues to open up and follow the community development models, as well as buyers updating their buying guidelines to prefer options that give you the choice rather than trying to lock you into a particular commercial offering. Customers look for reliable solutions; open source is the trustable, innovative and most sustainable approach to that need.

Several years ago SUSE's Linux Enterprise Server (SLES), the flagship enterprise Linux product from SUSE, was donated to the openSUSE community. If you use openSUSE Leap today, you are running the exact identical binaries to those available to SLES customers and the sources and all updates are available publically available via the same route. We've observed an uptake of usage of both openSUSE, as well as SLES as part of that.

Hannah: Should people be worried about what's going on?

Whenever something surprising or worrisome happens, it is a good idea to look at what your values are and what new choices to make going forward. SUSE has added with their announcement a choice to the available options on the table. Everyone has the freedom to choose, and if in doubt, I think people should choose what gives them the most freedom. Publicly available, open source gives you a freedom that restricted or proprietary software cannot provide; and it comes with the backing of decades of experience behind it that has shaped "enterprise linux" since late October, 2000.

Hannah: Dirk, you’re the voice of reason! We almost always have a choice, and change is hard, but it’s always worthwhile to choose freedom. Thanks for putting my anxiety to rest.

Now let’s talk about how you’re making the experience of using open source better for everyone. You previously built your own CDN to manage your download infrastructure. Can you explain how you did that? What were some benefits and drawbacks to that approach?

Dirk: In the beginning, there were a few dozen (now a few hundred) mirrors of the openSUSE distribution on community sponsored mirrors (university FTP sites and similar). These are pulling new content at regular intervals via rsync from a staging server that the openSUSE infra team is operating. Users had a web page where they had to pick a mirror from and then hope their choice of mirror remained updated and available. That hope was not always justified.

Eventually openSUSE users were loud enough so that an effort to address those issues started. We realized that these community-provided mirrors can be partial mirrors (e.g., the site admin by themselves decided to not mirror some parts of our trees), outdated or even occasionally down or overloaded, so a former colleague started a project called MirrorBrain which got later revamped in a new project called MirrorCache. Both are basically a two-tiers open source framework to use a few trustworthy, reliable redirector proxies to HTTP redirect to the existing, partial, and potentially outdated or unreliable mirrors all over the world. For that, it continuously keeps track of which mirrors are currently available, what part of the content they serve and how up to date they are. When a client is contacting the main redirector, it does a client IP geo-lookup and redirects the request to a nearest available mirror that has the file.

This setup has served us fairly well, and even with the addition of the Fastly CDN remains in place for the foreseeable future. The addition of Fastly is going to help us in three areas:

We more frequently ran into a situation when a client was requesting a particularly new file, there was no mirror available (because it hasn't been mirrored yet by any of the mirrors that we do not have direct influence over) or no good mirror nearby. In both cases we ended up serving the file from the main redirector, which took away the bandwidth that was shared with the mirrors who tried to fetch the updates themselves, so that we could leverage the mirror to offload from our bandwidth use. This often caused hours-to-days-long problems when particularly large snapshots of our premier rolling openSUSE Tumbleweed distributions were released, as users jumped on it before any mirror caught up on it.
We realized we have in some areas of the world an abundance of fast, reliable mirrors, while in other areas where we have emerging users we have no mirrors at all or only somewhat limited mirrors. Fastly's large and well balanced CDN network is going to help us with serving those users better.
Due to the growing popularity of the project, our primary redirector proxies are about to hit scalability limits due to the number of incoming requests that they need to process. The caching features of Fastly's CDN will help us with both reducing the incoming request rate as well as cutting down their latency significantly.

We're in a cautious rollout of the CDN, so both systems are currently in use in parallel. We see the benefits of the ability to offload a lot of load peaks to the CDN and will continue to refine the usage over time.

Hannah: What prompted you to want to switch to using Fastly? Any improvements you've seen in performance so far? Any other benefits?

We've done a set of controlled experiments with the Fastly trial program from various locations on the globe, and Fastly's CDN helped us to reduce request-handling latency in several regions by a factor of 5 and more. Due to a variety of reasons, the "redirect latency" is very visible for end users, so we are confident that this leads to a better experience for our users.

Also, as it improved the availability, reliability and geo-distribution of our mirrors, for many "remote" locations Fastly provides a significantly better bandwidth for our users when files are served from the Fastly CDN cache directly, although those effects are harder to quantify, in many locations that improved transfer times by about 20%, with bigger effects in locations where we, so far, had no mirrors at all.

Hannah: Twenty percent improvement! That’s great. I can’t wait to hear what you build on Fastly next. Where should people go if they want to get involved with openSUSE?

openSUSE.org has links on how to get in touch with the openSUSE community. news.opensuse.org is a good place to start with just following the news of the project. Coincidentally it is 18 years of openSUSE project this month :).

—

Happy birthday, openSUSE!! Cheers to 18 more years of free as in freedom.

Now let’s go build the good internet — together ⏩

I believe we can still save the web 💞

Hannah Aubry — Thu, 14 Sep 2023 21:00:58 +0000

In today’s digital age, trust has become a cornerstone of building a better Internet.

Preserving privacy for a greater good

The Internet was designed as a platform for peer research, not for the vast scale and diverse uses we see today. Over the decades, it’s grown in a way that has left it with many inherent vulnerabilities.

These vulnerabilities, not borne out of malice, were the result of choices made with limited information available at the time.

Fastly addresses these technological vulnerabilities by utilizing tools like Rust and WebAssembly. Leveraging WebAssembly’s sandboxing capabilities allows us to isolate potential risks, while Rust provides the memory safety essential for our modern internet applications.

Taming the human side

But the challenges facing the internet don’t just lie in its technical foundations. The societal aspects of technology, the human side, have grown equally unruly.

The trust deficit we experience today is palpable. People are wary of technology and its creators. Our major platforms, tools integral to modern life, are now used as vehicles for misinformation and chaos. A disconnect exists between those building technology and its end-users, often exacerbated by financial pressures and lack of proper oversight in the tech industry.

Despite this bleak landscape, there’s hope. We possess the tools to craft a better, more trustworthy internet. As we embark on a new era for the web, the foundation lies not just in what we build, but how we build.

The ethos? Openness, transparency, and collaboration. These principles drive better technological outcomes, ensuring an improved experience for all web users.

For many, open source stands as the embodiment of these ideals. While not all technology must be open source, perhaps it should be developed with the same spirit. Trust, after all, is foundational to the open source model. It hinges on community faith in institutions, and this trust, once broken, is painstaking to restore.

Transparent approach

Sharing a couple of personal experiences from my work in the open source community illustrates this concept. When I joined Fastly, I had the chance to recommit $50 million towards supporting open-source projects during the challenging times of the pandemic.

However, bureaucracy and inefficiencies had muddled our vision. The decision to pause our program in order to reimagine it was difficult and led to criticism. Yet, by taking a transparent approach, seeking feedback, and re-building collaboratively, we restored our trust within our open source community, Fast Forward.

Another enlightening experience emerged from my home instance, Fosstodon, on the decentralized social web (aka The Fediverse). Recently, a controversy arose due to their English-only policy, seen by many as exclusionary.

However, in response to the community’s concerns, Fosstodon’s administrators put the policy to a public vote. By opting for an open and collective decision-making process, they were able to retain the trust of their vast community, including mine.

Improved experiences

These stories serve as potent reminders. Trust in the technology realm is essential, and achieving it requires an open-source mindset. This approach calls for inclusion, clear communication, and cooperation. By granting all stakeholders not just a seat but a voice at the table, we create superior technology for users, preserving our communities’ vitality and improving overall web experiences.

Our ultimate goal is straightforward: develop both technical and human-centric systems that are intrinsically beneficial and trustworthy. From the core programming to the communities we build, let’s champion an Internet that stands as a beacon of trust and collaboration.

It’s been a wild week for the open social web

Hannah Aubry — Fri, 07 Jul 2023 22:53:27 +0000

Eight months since the Migration started and this week has been one of the most eventful for the Fediverse since. In case you missed it, Threads, by Meta, was released in the US, without ActivityPub support at this time, and has already amassed over 30 million users in the space of a couple days.

And on top of that, Twitter introduced view rate limits to "address extreme levels of data scraping and system manipulation". The rate limits caused a surge in new account sign ups and a huge increase in traffic to Fediverse instances over the weekend.

Fastly is committed to helping the Fediverse thrive. We care deeply about the open web, and are working directly with oss projects like Mastodon and Kbin, instance admins, and more to help them scale.

Nobody knows exactly what Threads means for the Fediverse, but we all know it means new challenges for scaling the open social web. So here’s what we know we can do right now:

For all Fedizens:

Don’t panic! Very little is known at this time about how Threads will implement ActivityPub, or what federating with Threads will look like, so stay engaged and thoughtful, but don’t worry too much. We can focus instead on what new possibilities exist.
Be kind. This is a scary and exciting time for the Fediverse. It’s been almost a year with very little rest for the developers, system administrators, and community managers who keep it running. The people building this place are tired, so be nice. They’re carrying a lot.
Lend a hand. There are so many ways you can contribute to, and help build, the social web. If you can code — great! But that’s far from the only way to contribute. You can volunteer or donate your time or money to your instance admin or to a Fediverse project like Mastodon, Kbin, etc.

For users:

Vote with your feet and voice. I love this recommendation from Lily Hay Newman at Wired — Don't Join Threads—Make Threads Join You. If you have concerns about the way Threads is building for the Fediverse, let them know and don’t sign up. Or if you have concerns about the way your instance admin is talking about Threads, move to a new instance. That’s what’s so amazing about the Fediverse; you are in control.

For instance admins:

You’re not alone. If you’re an instance admin that’s worried about the firehose of traffic and new users that federating with Threads will bring, just know that you’re not alone. There’s a supportive community of server owners in the Mastodon Discord (access to the Discord is a perk for Patreon donors), a network for Fediverse developers with a Matrix chat, and millions of people across the Fediverse to help and support you.
Write a federation policy and communicate it. If you don’t have a repeatable policy about the kind of instances you will and won’t federate with, now is as good a time as yesterday to write one. (Hachyderm published their thoughts on how they choose who to federate with today, and Mastodon has their Server Covenant too. Great policies to be inspired by!)
Let us know if we can help. Shameless plug, but if you read this you probably know that I lead Fast Forward, Fastly’s open source support program. Fastly is built to handle the traffic surges that come with scale! And we’re already helping some of the Fediverse’s largest instances.

Just some quick thoughts from me to close out a stressful, busy week! Get some rest this weekend, remember to drink water, and be excellent to each other. :)

Community Spotlight: Una Thompson is Making the Fediverse More Manageable with Jortage

Hannah Aubry — Thu, 11 May 2023 16:29:23 +0000

Decentralization is what makes so many of the Fediverse’s unique features possible. And it can also make running an instance totally unmanageable. But Fast Forward program member and Jortage maintainer, Una Thompson is using Fastly to help solve one issue that arises with decentralization: content duplication. We wanted to learn more about the issue and how Una is solving it, so we sat down with her to learn more.

The Fediverse is a wonderful place — it enables small but interconnected communities to moderate themselves and choose with whom they associate. It gives users the freedom to own their data and migrate between instances. But with the freedom that decentralized social networks provide, there are also tradeoffs.

Federation ends up flinging a lot of content, i.e. media objects, back and forth between servers. Whenever someone posts, their instance pushes the status and objects to the instances of everyone who follows the poster, and all those instances download it. Each instance then separately uploads to its storage provider, duplicating it by the number of the poster’s followers.

It’s a tough problem, and Una has solved it in a really neat way with Jortage. It's a communal project that provides object storage and hosting and deduplicates object storage for pool members, reducing everyone's overall storage costs. To ensure media stays fast despite the project being used on over 70 instances all over the world, Jortage utilizes Fastly to accelerate and cache downloads. With Fastly, Jortage has avoided investing in their own global servers, keeping costs even lower, and makes the experience of cruising #CatsOfMastodon faster and more engaging. :)

Hannah: What initially inspired you to create Jortage?

Una: It's one of those things that just kind of comes to you while doing something else; at some point, I wound up thinking about the way media federation works and had a "hmm, there's no way that'd actually work" idea, prototyped it, and it worked great. So I scaled it up and got some other instance admins in on it, and the rest is history.

At one point I got a DM from Gargron [Eugen Rochko, CEO of Mastodon] asking how on earth Jortage actually works, as FFmpeg and ImageMagick (used by Mastodon to process incoming media) are nondeterministic — honestly, I don't know. We shouldn't be getting results this good. :P

Hannah: So how does it work?

Una: The basic way the storage pool works is that every uploaded file is hashed, and a MariaDB database stores a mapping of hashes to paths; new files are then uploaded to our backend storage provider, Wasabi, using the hash as a filename. Requests to pool.jortage.com look up the path in the DB, and if a match is found, return a redirect to blob.jortage.com with the relevant hash. Pool points to a server I run with poolmgr, and blob just points directly to Wasabi.

The storage pool CDN connection was pretty simple. The most complex thing in my Fastly config is the song and dance involved in doing a redirect for the root path, and it's highly specific to the storage pool. I also had to enable Segmented Caching to make Jortage work with videos, as Mastodon’s upload limit is 40MB.

The software that powers Jortage is the custom poolmgr. It’s very much a “standing on the shoulders of giants” project. Without S3Proxy and Apache JClouds, it would've been a much much tougher project to pull off. So poolmgr exposes an S3-compatible API that Mastodon (as well as Pleroma, Akkoma, Misskey, etc) talks to. However, the vast majority of storage on a Mastodon instance goes to its S3 provider, so ~70 instances are feeling the benefit of Fastly regardless. (Some rough benchmarks I did while waiting for DNS propagation suggest that Fastly’s total download time is 2-5× faster than our previous vendor, including uncached requests.)

The heaviest parts of Mastodon can't really be solved with a CDN; things like the job manager (Sidekiq) and federation requests are actively made worse by a CDN — just look at all the instances running behind Cloudflare that regularly have mysterious federation issues. Media is all the big files, though.

Hannah: The heaviest bits of federation — do you think there is a solution to make federation requests and Sidekiq run more efficiently?

Una: I don't think there's any web-based service you can point to today and call "efficient" without a lot of asterisks. "RESTful" services that sling JSON back and forth are not efficient by any definition of the word. I'm certainly not an expert on ActivityPub, but in general, it’s not what you would call an efficient protocol.

Making a single post causes your instance to send that post to every instance that's following you. That, in turn, causes those instances to make a request back to your instance for the thread context, which then can result in more requests for posts. They'll also check for your current profile information, which will cause them to load the links in your profile looking for rel=me links to process verification. And, of course, if there's attachments, they all get downloaded. All of these create jobs in the Sidekiq queue, which in the default Mastodon config has a very low number of threads and runs in a single process. It's not even well-documented that multi-process Sidekiq is possible — a lot of people heard of for the first time via this blog post by Nora Tindall.

Of course that itself adds overhead (Sidekiq uses dizzying amounts of memory), so I've been interested in the TruffleRuby solution to keep it all in one process and provide a better GC.

If that back-and-forth request explosion could be replaced with eager data pushing in the initial federation it'd likely help. There's another Jortage project related to the link resolution issue, called jort.link.

Some of it has to do with the fact it runs on the Ruby interpreter, and one of my "whenever I get a chance" projects is to try running Mastodon under GraalVM via TruffleRuby. I feel this will be a major win for Sidekiq, as it introduces JIT compilation and true multithreading. A global interpreter lock makes parallelism very shaky in vanilla Ruby (also an issue with vanilla Python, and one of the main benefits of PyPy).

The centralization inherent to Jortage is a negative for sustainability but running truly independent services that scale costs a fortune.

Hannah: What are your tips for other devs who want to set up an instance?

Una: Think long and hard about it. Running an instance is setting up a community, and running a community is a long (essentially endless) commitment, and doing proper moderation is very hard. I run a closed instance, and you don't have to look far to see all the stories of people who went into running a public instance without being prepared for what that means.

If you want to learn more about Jortage or connect with Una, visit Jortage’s homepage and reach out on Fedi to say hi.

Mastodon has 10 million users! 🎉

Hannah Aubry — Tue, 28 Mar 2023 13:31:08 +0000

Here’s what it will take to get to 20 million.

Let me start by saying that I’m a relative newcomer to the Fediverse, as you can see on my profile. Many people (and institutions, NGOs, governments, and more) have been thinking about the space—and the technologies underpinning it—for much longer than I have. Most of what I’ll be doing here is pointing to and surfacing their fantastic work.

What I know really well is community and communal effort. Through my work managing Fastly’s open source program, Fast Forward, I spend most of my time learning about and supporting some of the world's most successful and long-standing communities. They work together, often across time zones and languages, sometimes with no incentive besides their passion and determination to improve the world or solve problems.

The Mastodon and Fediverse community is incredible. (I’ve tipped my hat to Mastodon's community before.) I mean the people who gather in these spaces to talk, share, laugh, and learn together, and of course, those who build and run the servers, platforms, and software we use to navigate the Fediverse.

But open source technologies and communities fragment. They’re supposed to. It’s the direct result of the freedom open source represents. Someone decides they don’t like the direction of a project, so they fork it and set off in a different direction. It’s wonderful, but fragmentation is a pitfall too. It can hamper progress and reduce impact.

The technology and protocols of the Fediverse are decentralized — our efforts to build and improve them don’t have to be.

As the community and user base continue to grow, we must be deliberate in staying organized and working together. We need to share ideas, solutions, and resources. Not to stave off fragmentation completely but to use resources efficiently and to make our work better. There are brilliant minds in this community and many passionate people who want to fix existing technical and policy issues. If we work together, we can solve them.

Now to cover what needs to be improved and fixed.

A few days ago, I posted on Mastodon asking my friends there to share what they thought needed to improve to grow the Fediverse. It was a great thread; I have really smart friends! Before I get into their thoughts and share some of my own, I want to address a comment I got a couple of times because it’s a valid point, and I want to be clear about my intentions.

I don’t have an agenda here except to be helpful however I can. My agenda is not growth for growth’s sake. I don’t think that’s good or desirable. I’m excited by the idea of more people finding the community and kindness I've found on the Fediverse too. (I haven’t felt so at home in an online space since I got my start in community posting on the old Ctrl+Alt+Del webcomic forums.) I think the community-first and welcoming nature of the Fediverse can be maintained, even at a greater scale, if growth occurs thoughtfully and intentionally.

To me, that’s why we need to make changes and improve — to build systems and spaces that are safe, trustworthy, and delightful by default, so scale isn’t scary.

Here’s what my fellow fediversizens (let’s workshop that…) thought about growing the Fediverse, along with some of my thoughts and links to great work that is in process, and tangential or related work that could inform the Fediverse’s future.

Think beyond Mastodon

Rather fitting this was shared by someone posting on Pleroma 😂— it’s so true! Mastodon is the most popular software for navigating the Fediverse, but it’s not the only one, and it’s not based on the only federated web protocol (ActivityPub) either. You can check out some of the other software and platforms on Fediverse.party. (And if you don’t know how the Fediverse works, you can learn about it on JoinFediverse.wiki, or check out this post from our company blog (there’s info about really neat projects being forked and built on top of Mastodon itself in there, too).

Actually, one of my favorite things about the Fediverse (did you know RSS & email are part of the Fediverse too?) is Webmentions. You can see it in action over on the Open Source Initiative’s blog, Voices of Open Source, here.

Verification

I think this is crucial for the Fediverse to become a “trusted news source” in the same way Twitter was, or for journalists, politicians, and other people in the public eye to feel safe on it. They need to be able to prove that they are who they say they are, to give their followers reassurance that they’re authentic, and to disavow impersonators. And what’s more, we all need to trust that system for it to work.

I’m unaware of any work being done specifically in this space for the Fediverse (if you are, I’d love to hear about it!). My teammate Keith Kurson told me about a new project happening in California right now called the Digital Identity Project. There’s a broader effort, supported by many key players in the tech space, ID2020, to promote the use of digital IDs too. For those who want their Fediverse identity to be tied to their IRL identity, linking their digital ID could be an option. But this option might be years out before it’s feasible to implement. What can be done now?

Community Moderation

The moderation issue is crucial to any public digital and social space. And the capacity for the community to moderate must keep pace with the size of the community. My mind immediately goes to Facebook’s (now Meta’s) failure to moderate their communities at scale. In some countries, they don’t have enough, if any, human moderators that speak the local language or understand local customs or idioms. We can’t let that happen here.

I’m really excited to learn more about what Jaz is working on. (Reach out if you’re interested too — and let’s support the work as we can!) Education for moderators and instance admins will be crucial. A wonderful aspect of the Fediverse is that moderation scales alongside the community because each instance admin takes on the responsibility of moderating their community. And users can switch instances if the moderation policies don’t suit them.(That being said, I know the recent explosive user growth of users has been challenging for many admins. If you can, donate to your instance or volunteer as a mod!)

And I wanted to mention Izzie’s comment here because it’s a key failure of Meta’s scaling too fast without intention. We can’t trust AI (yet, if ever) to do this work for us. AI reflects the dataset it was trained on, and therefore our systemic and societal biases. We must have human controls for moderation that are based in local communities and have knowledge of local languages, customs, and events.

Trust and Community

It will be interesting to see how local and national governments, educational institutions, NGOs, and companies continue to show up in the Fediverse. Many examples of those types of organizations are already spinning up Mastodon instances. I will list them here (let me know if I missed any!)

The German federal government: https://social.bund.de/about
The European Union: https://social.network.europa.eu/about
The City of Amsterdam: https://social.amsterdam.nl/about
SURF, the Dutch IT cooperative for education and research: https://social.edu.nl/@SURF
Internet Archive: https://mastodon.archive.org/@internetarchive
Mozilla: https://mastodon.social/@mozilla
Medium: https://me.dm/@medium
Flipboard: https://flipboard.social/@Flipboard
Vivaldi: https://social.vivaldi.net/about

The presence of governments and such organizations on Mastodon further legitimize the platform. They represent a trusted communication channel for their citizens, constituents, and fans by linking their web and social presence. And they help build trust in representatives of those institutions by tying the individual’s Mastodon presence to the organization’s instance.

What’s more, these instances serve as community gathering points. I’m imagining a future where city governments (or beyond!) could offer community groups, in addition to civil services, a profile on their instance so that the instance’s local timeline becomes a gathering place for the community to share resources, events, and more — a true, digital town square.

Discoverability & Growth

I heard from several people that they’re having difficulty finding people and content that interests them. And some people they’d really like to see on the Fediverse aren’t there yet. As Danilo points out further down in the thread, there are some growth-hack tactics that could be used to help bring entire communities over to the space. “If you want specialized groups to stick it helps to give them a shared onboarding experience, I think. Then they know they’re in it with other people, they can compare notes, etc.” (It’s been really cool to see some communities migrate en masse, like infosec.exchange and hachyderm.io.)

I think one answer to getting people and communities to join is simply the passing of time. As the platform grows and stabilizes, more people will join. In the meantime, there are things we can do to make the Fediverse even more welcoming and help keep people coming back. If you see someone who’s just joined or is confused, say hi and offer your help! It only takes a moment and it just might make their day. 😊

Here are resources or tips I keep suggesting when I do so:

I think Fedi.Tips is the best crash course out there, and you can follow them here: @feditips@mstdn.social
Follow, and use, hashtags for content that’s relevant to your interests.
Go check out the blogs of your friends, favorite news sources, etc. Maybe they’re already here!
If your local timeline isn’t interesting to you, find a different one and migrate. There are instructions to do so on Mastodon’s site. Joinmastodon.org has a directory too, and fedi.garden keeps track of well-moderated instances.
Ask your friends who they follow, and boost people and content you find that you like.

It’s a fraught issue due to privacy concerns, but search is one way we could address the discoverability issue. My colleague Anil Dash wrote a blog post about how we could build a search that the Fediverse would welcome. There’s an existing opt-in search option called Tootfinder.ch too.

Features

There are features that are missing that will make people feel more comfortable, and help give them a more delightful experience. BatManitou covers two I’ve heard frequently from many people: quote posts and secure messaging. There are inelegant workarounds for now (like linking to the post you're commenting on in a new post, and moving private messages to an encrypted platform.)

On the other hand, the fact these features aren't included, or the way they are presented (like with private messaging), are deliberate choices. And to me, they are wise ones. I don't want the Fediverse, or my home on Mastodon, to become a place where dunking on people is systemically and technically incentivized or encouraged. And the way private messaging is presented serves to remind you that your messages are not encrypted. Platforms don't need to do or be everything to everyone, which is what makes the Fediverse great. Mastodon may not have quote posts, but there is a protocol for it in ActivityPub! Which Pleroma and Misskey have implemented.

Beyond functionality features like the ones above, there are so many great and fun things being built for the Fediverse. Shameless plug because I'm on the Glitch team, but there are a lot of things being built on Glitch.com that are very useful and delightful. You can check out a round-up here.

I’ve probably missed some areas we could grow and improve here — what do you think?

By the way, in this post, I’ve focused mainly on the more user-oriented priorities and growth opportunities for the Fediverse. In a future post, I’m planning to address some priorities that we need to address further down the stack, like DDoS mitigation and resource pooling with an eye towards sustainability, and reducing strain and stress for instance admins and their servers. I'd love to hear your thoughts about that!

Finally, I’m really excited to be participating in FediForum March 29-30, where we’ll discuss all of these topics and more. Hopefully, I’ll see you there!

Chat me know in the comments here or message me on Fedi: fosstodon.org/@haubles

Welcome Mastodon to Fast Forward!

Hannah Aubry — Wed, 22 Feb 2023 21:48:41 +0000

We’re thrilled to announce that Mastodon is now a member of Fast Forward.

We believe things are better when they’re a little closer to you — when you have a say over your data, your safety, and your community. If we’ve learned anything over the past few years, we’re stronger when we stay connected.

Welcome Mastodon to Fast Forward

We are super excited about this project because of what Mastodon represents — not only the power of community but also the power to build communities, to own and control your data, to leave a digital space, and take your data and your friends with you when and wherever you go. Helping a project like Mastodon is why we have a program like Fast Forward in the first place — it truly delivers on the promise of the open web that drew so many people to ActivityPub and the fediverse.

In some ways, the motivation that led Eugen Rochko to create Mastodon is similar to what led our founders to create Fastly. They found something about the internet that was broken, so they set out to build something that would fix it. And the community that Eugen has helped lead around Mastodon has succeeded beyond anyone’s wildest imagination.

At Fastly, we can help the open web thrive by providing scale and security. And we can help an open-source project whose biggest challenge is being available for, and welcoming to, everyone around the world who wants to take advantage of this new platform.

How we’re helping

Mastodon was created long before the most recent news cycle, but it has certainly seen people joining at a much faster rate over the last few months. That means more people hitting the joinmastodon.org domain, more activity throughout each 24-hr cycle, and more load on the mastodon.social and mastodon.online instances.

The increased attention has been both positive and negative. On January 31st, Mastodon and its flagship instances were hit by a significant DDoS attack.

To understand the magnitude of such an attack, we generated a graph representing the relative trend between average traffic to joinmastodon.org and the requests per second (RPS) during the attack.

What’s more, application-layer (Layer 7 of the OSI Model) DDoS attacks like those directed at Mastodon are typically very short-lived, meaning it was difficult for the Mastodon team to react in the moment. Right now, the team administering the mastodon.social and mastodon.online instances, as well as the rest of Mastodon’s infrastructure, such as joinmastodon.org, consists of Eugen, Renaud Chaput, and the excellent community of contributors, sponsors, and instance admins that support the nonprofit. (By the way — Mastodon is hiring a full-time DevOps Engineer.)

And while that might work for one-off attacks or uninspired hackers, anyone persistent enough can easily switch tactics to get by those quick fixes.

Fastly absorbed the attack on joinmastodon.org — we’ve got automation to respond to such attacks. Given our place as a frontline operator sitting between systems and the people that use them, we’re constantly monitoring the threat landscape, evolving our defense and mitigation stance, and, what’s more, empathizing with victims. While DDoS attacks continue to grow larger and faster, the industry’s ability to respond is growing even faster still.

We’re glad to be able to protect Mastodon and to be an option for anyone else who wants to scale and protect their community with integrity (see our trust and community policies).

We’re also mindful of not wanting to centralize the infrastructure of a social network that is decentralized by design. Mastodon can easily migrate to another CDN or a multi-CDN approach. Put simply, we’re handling the network — delivering and securing content — so Mastodon can handle the stuff that matters most to them — encouraging healthy communities with effective community management and brilliantly creative participants. Because we believe…

The future is federated

Mastodon is a growing project and community, and they need help. We already mentioned that they’re hiring a DevOps Engineer. They’re hiring a Software Engineer and a Product Designer too. You can also join the community discussion, suggest a feature for their roadmap, and get set up to contribute to the codebase.

And Mastodon is one platform within a world — a fediverse? — of projects, platforms, clients, and apps built to intercommunicate. It’s a whole fediverse.party! We can’t wait to see all the new and innovative stuff that’s yet to come.

Speaking of, are you building for the fediverse? We want to support you! We’re already supporting some incredible projects building for the new web (I can’t wait to tell you about them in the coming weeks!), and we want to support you too. If you need help scaling your project, apply to Fast Forward here. Or if you want us to signal boost your project, need advice, or want to show us what you’re building, reach out to us on fedi!

We can’t wait to see what you’re working on! Now let’s go build the good internet — together. ⏩

Why We Love Rust: Ferris Is Only Part Of It

Hannah Aubry — Fri, 17 Feb 2023 03:36:53 +0000

Danilo Campos and I co-wrote this post!

While you’ve probably heard plenty about Rust—its benefits for correctness and memory safety, perhaps, or its adorable unofficial mascot, Ferris the crab—what you may not know is that the project has set a high bar for its onboarding developer experience. There are lessons any project can take, and we wanted to walk through them.

Automate the knucklehead stuff

Computers are our friends! Whenever possible, it’s great to use that to make it easier to get moving with a new technology. In Rust’s case, the getting started page detects the visitor’s operating system, then uses that information to provide relevant setup instructions.

It looks like you're running macos, Linux, or another Unix-like OS. To download Rustup and install Rust, run the following in your terminal, then follow the on-screen instructions. See "Other Installation Methods" if vou are on Windows." Followed by the curl to install rustup"/>

From there, an installation script sets up the Rust tools with just one line of input. Instead of a long sequence of instructions, a tidy piece of automation handles setup.

Finally, there’s even a one-click Rust sandbox where you can skip setup altogether and dive into a pre-built environment to start playing around with the language.

Fast time-to-hello-world

Rust knows many of its practitioners have existing experience with tools they love. Integrations between Rust and popular IDEs get prominent notice, so you can quickly get moving in a familiar environment.

Whether you prefer working with code from the command line, or using rich graphical editors, there's a Rust integration available for your editor of choice. Or you can build your own using rust-analyzer.VS CODE. SUBLIME TEXT. ATOM. INTELLI IDEA. ECLIPSE. VIM. EMACS. GEANY."/>

Once you’re set up, Rust walks you through the mechanics of setting up your first project, explaining context along the way. With just a few commands, you’ve got running code that prints “hello world!”

Streamlining the process to get to working code, even when it’s just a simple program to print a friendly message, is essential to developer success. It makes it clear that the whole toolchain is working correctly, setting the table for productive exploration of the new technology.

Connection to the larger ecosystem

Rust has a lot of power, but like any ecosystem, it has specific details you’ll need to understand to make the most of it.

While Rust has abundant docs, the Getting Started page elevates essentials like package and dependency management front and center, since they’re a topic every developer will need to understand to be productive. Thus introduced, Rust then walks you through how to use these dependencies in your own code. This unlocks the larger universe of existing Rust code, letting you build what you want without having to reinvent the wheel.

So how’d they do that?

We wanted to know! So we asked, and Carol Nichols, author of The Rust Programming Language, answered our questions.

Rust's developer experience is optimized for rapidly getting developers up and running — how did the culture develop? How do you prioritize, emphasize, incentivize, or invest in it?

Many people come to Rust with experience in other languages and ecosystems, and they’ve used that experience to improve upon the status quo. For example, many people wanted a systems language like C++ but with the package management and build tools from Ruby that make sharing libraries easier. There have been cases when developers start using Rust and find friction or confusion. Caring folks in the community try their best to not get defensive, but to take that as an opportunity to improve sharp edges that we’ve gotten used to. Another is Esteban Kuber, who actively solicits error messages that the compiler gives that people have found confusing then improves them.

What advice do you have for others looking to improve their developer experience?

Do some usability testing! Create a task like following a tutorial, give it to someone in your target audience who you’d like to improve the developer experience for, and watch them (in person or remotely) try to accomplish this task. Write down every spot they have trouble, and only help if they’re truly stuck. Then turn those problems into improvements to tooling, documentation, or features and repeat!

Where should people who want to get involved in Rust's developer experience go?

It depends on the part of the experience you’re particularly interested in! Here are some I can think of — The Dev Tools team, for work on:

Cargo
IDE and editor support
Clippy (Rust’s linter)
Rustfmt (code formatting)
Rustdoc (the tool that generates Rust documentation)
Docs.rs (crate documentation hosting)
Rustup (Rust version manager)
Bindgen (tools for generating FFI bindings)
Rustfix (a tool that can automatically fix some warnings)

The Compiler Team, especially the Diagnostics Working Group that improves compiler error messages. The Libs Team, for work on the contents of the standard library documentation

But you don’t have to take our word for it!

Hit the Rust Getting Started page and try it for yourself. It’s a tidy example of how to set up developers for success. Maybe you can use these lessons for your own open source project!