DEV Community

Cover image for Project Benatar: Fending Off Data Black Holes
Ben Halpern Subscriber for The DEV Team

Posted on

Project Benatar: Fending Off Data Black Holes

In this post, we outline a new project which gives a name to some values we hold tightly at DEV.

First, an examination of the web we live with...

Warning: This post contains mixed metaphors and loose physics accuracy.

In a galaxy not so far away

The web started as a collection of small and medium-sized websites interconnected by links and discovered by search engines. But as the platform evolved, some of the more leveraged enterprises began to expand their power and consume more and more.

In the galaxy of interconnected entities, everything develops gravity. Gravity is a platform's power as achieved through immense traffic, social and economic value, and competition-defeating moats that strengthen over time. Every website has gravity if traffic and data flows through it and users orbit.

Some platforms have a lot of gravity. Google, Facebook, Twitter, Wikipedia, Reddit, etc. come to mind.

This gravity can be really good for users. Facebook's general mission of connecting the world is noble in theory. In the developer ecosystem, the gravity of StackOverflow and GitHub help software developers get so much done, and organize us through familiar destinations, tools and form-factors.

There is a natural gravity in any galaxy or solar system. But when any spacial body accumulates too much gravity it can become a Black hole.

As trillion dollar web companies become the norm and each of these platforms benefits from sucking in as much data as possible, they begin to swallow their own solar systems, destroying competition and imposing their will on everything that is left in their wake. They consume our data, they consume the open web, they make previously-open standards into their standards.

We don't have any special contempt for any platform in particular, but we have growing concerns about the healthy relationship between platforms and their orbiting users, developers, partners, etc. Google is keeping more and more of its traffic on its own servers instead of passing through to independent sites. We could go on and on with similar examples.

Data black holes are clearly a pattern. Organizations practically have a fiduciary responsibility to assume this role.


DEV wants to continue growing because the bigger we are, the more that we can deliver value to the developer ecosystem. But as we grow, we think it's important to be thoughtful about all of this, hence our ongoing commitment to this kind of thing.

A galaxy needs to be comprised of some medium and large planets, plenty of moons, and a few healthy stars. We feel a responsibility to play a small part in providing stability across our system, We don't want to grow to be a black hole of data, and we want to help diminish the gravity of any black holes in our space.

One element of that outlook is the ongoing development of the generalizable version of our platform. We hope that if a lot of folks stand up their own communities similar to DEV, it will create a healthy solar system of independent but compatible medium-sized planets that are each powered through their own incentives. The diversified ownership structures will prevent the consolidation of power, a key component that leads to black holes.

We try not to have delusions of grandeur, but we believe in the power of open source and we work every day towards some of these ideas as part of our longterm vision. We've been writing here and there about these ideas since before we even went open source last year.

We are planning to release the generalizable version of our platform at some point this year.

But first, introducing a smaller project

Supporting the growth of full-blown platforms similar to DEV is an exciting future priority. But not every site should aspire to serve such a sweeping mission and broad set of stakeholders.

Thus, a key element in this grand initiative is to support the development of the smaller platforms that can interact with the large and medium sized ones (DEV and others). What this means in practice is that getting data in and out must be simple; determining canonical urls and other logistical aspects of content ownership must be straightforward and understandable; and the process of transportation between different planets (DEV, Medium, future DEV instances, etc.) must be fluid and simple.

Introducing Project Benatar

Benatar is the name of the ship in Guardians of the Galaxy. We feel like this reflects the energy of this project. That ship is actually named after singer Pat Benatar.

#projectbenatar

Project Benatar is a mission to create tooling that supports the development of independent platforms and compatible instances. It is an undertaking that will make it easier to stand up an independent site that is compatible with DEV, related instances, and other platforms to the extent that we can create integrations (Medium, Twitter, etc.). It is a project in developing APIs for creating, updating, and deleting data — as well as exporting it in useful and relevant formats. All of this functionality is connected to the ultimate goal of providing more control to the end-user and content creator.

Because this is related to the underlying mechanics of the DEV platform, we will work to help create all the relevant APIs. We’ve already developed some functionality in terms of our publishing API, our data export functionalities, our RSS publication functionality, alongside the rest of the tooling. But this is a long-term project, and we will seek to make our role in this movement as useful and mature as possible.

In essence, we want personal websites to have a more powerful role in the ecosystem, while we all still benefit from the value that community platforms create.

This project will build on principles of POSSE and decentralization, but mostly centers around these ideas can influence our community, ecosystem and projects. As it grows, it could take on broader ideals, but we want to be practical about it. This will be an effort in reducing complexity of these concepts and creating the best possible user experience.

POSSE:

POSSE is an abbreviation for Publish (on your) Own Site, Syndicate Elsewhere, a content publishing model that starts with posting content on your own domain first, then syndicating out copies to 3rd party services with permashortlinks back to the original on your site.

As an indication of the gravity we have already generated, this is actually a project we will not be taking on ourselves, but will be launching this initiative with partners who believe in these values, and have products which already support these ideas.

We’re delighted to be joined in this campaign by our friends at Stackbit, a startup that specializes in making modern website stacks accessible and straightforward. They are already building software in this field, and we felt that combining our efforts through a shared initiative would benefit the whole web.

Stackbit will be publishing about the work they are doing and and already have been doing in this space. Here is a post related to their existing efforts to help developers deal with their Medium data/presence:

You may recall that we had some recent opinions on the Medium in particular:

In general, Stackbit will be bringing an expertise in this space, and we are very excited to see where this goes...

Next steps for you

If you are interested in supporting these ideas, please feel free to do the following:

  • Follow the Stackbit org on DEV
  • Follow the #projectbenatar tag on DEV
  • Partake in the discussion and contribute to our open source initiatives in general

As we work on these ideas and come up with new launches, this will truly be a community initiative. If your organization feels like they are positioned to get involved in this project in some way, feel free to reach out to us. My DEV inbox is open. 🙂 In general, partnership ideas of any kind can be inquired about at biz@dev.to.

We're not sure what will come of our efforts, we just know that these are ideas worth working on.

cover image credit: marvelcinematicuniverse

Happy coding!

Top comments (26)

Collapse
 
awwsmm profile image
Andrew (he/him)

We hope that if a lot of folks stand up their own communities similar to DEV

So like Mastodon meets StackOverflow?

Collapse
 
ben profile image
Ben Halpern • Edited

That's a good way of putting it.

It's definitely Mastodon meets something. Another lines of thinking is a decentralized Reddit where DEV is like /r/programming and /r/javascript is also a thing but so is /r/nba and /r/torontoraptors etc.

All of which could be stood up independently by entrepreneurs or general community builders and their place within a broader mesh would come down to shared ideas about community moderation principles etc. Of course, as everyone knows, effective and inclusive moderation is core to our whole project and we'll always want to be leaders in this aspect.

The key is that incentives are aligned, and the value creation goes as far as possible. It's part of the magic of open source and we're just trying to build cool things.

There are also use cases for private networks (e.g. company intranets) as well as a possible replacement for hulking Slack communities etc.

Project Benatar sort of sets this up because the connective tissue between the programming community and the basketball community is your website. How profiles are shared across spaces etc. is something we'll figure out.

Our project has always been super longterm-oriented, but as we expect to release some of the pieces within this year, the shape of the whole idea could start becoming clearer.

Collapse
 
awwsmm profile image
Andrew (he/him)

What's the differentiator between the DEViverse and Mastodon in that case? A push toward universal, open protocols of communication between social networking sites? Or do you see DEV as expanding beyond social networking and blogging?

Thread Thread
 
ben profile image
Ben Halpern

We're not necessarily trying to define what we do in terms of how it differentiates with other approaches, but we feel like our efforts of consistently scratching our own itches in terms of building inclusive community and making the content on the platform have initial impact as well as lasting values are things we hope others can replicate with the tooling we've built.

I think Mastodon took a more bottom-up approach where the ecosystem was designed to be federated from day one and the choices were always centered around that goal. I think this leads to different UX choices and userbase relationships along the way. Who knows, the ideas could converge, but for now we're happy to be plugging forward with solving our problems as they arise and hopefully create a situation where the software we built is as open and re-useable as possible.

There was another thread comparing and contrasting DEV with HashNode and I think the comments were right to ultimately conclude that DEV didn't really differentiate strongly for the better strictly in terms of platform but we've always had good ideas in terms of community building. As community building helps us ultimately create better software we create a positive reinforcement loop for the better.

Or do you see DEV as expanding beyond social networking and blogging?

The answer is probably yes in that I'm not necessarily trying to draw lines of where this project goes, but I'm not entirely sure what you mean here. Could you expand?

Thread Thread
 
awwsmm profile image
Andrew (he/him)

I think I was thinking of along a more narrow "product differentiation"-type vein, whereas you might have a more grandiose vision. This comment...

Love the POSSE adherence.

To me, federation is key (and the reason email is so popular). Is there such a thing for spreading content semi-automatically?

I'd expect either HTML or Markdown as a basis, and wonder how that would play with embeds and custom tags.

...might better outline what you're thinking? DEV wouldn't be just a social media site, but rather sort of a standardised mode of communication? Or maybe I'm extremely confused and should just wait for the press release 😂

Collapse
 
waynejwerner profile image
Wayne Werner

Though probably more like SO blogs or something, since SO is for questions and answers, rather than discussion. This platform (like Medium) is definitely more for discussion.

I do love the premise here, though. It's actually one of the reasons that I've started to try and excise myself from the gravity well of many of these huge systems.

If you look at when the web was young, and Google embraced and supported things like RSS and other open protocols. They've tried many times to kill off and destroy those parts of the open web, to say nothing of Facebook (and how incredibly ironic is it that they've introduced their own cryptocurrency? The only more bizarre organization I could think of issuing crypto would be the US treasury).

I've been working to get back to open protocols like email and RSS/Atom, among others. Storing, rather than streaming.

The other thing about these massive bodies is that they have a commercial necessity to bring you back in, so notifications become a huge part of people's lives. I noticed that I was living in a notification-heavy world, so I started dismantling my notifications, keeping them only for the actually important things.

My life feels a lot more calm, and I've created content more often, by quite a bit.

I do like the idea of federation, like Secure Scuttlebutt, or Mastodon, but so far what I've seen about federation is that there's still some lack of control, as well as in some cases a lack of... I don't know what you'd call it, but basically - we have multiple devices usually, and we need better tools to be able to bring our content(s) with us. Without necessarily having to depend on a commercial service, aside from the availability of TCP/IP, or the Internet.

I don't have those answers yet, though :P

Collapse
 
wes profile image
Wes Souza

Love the POSSE adherence.

To me, federation is key (and the reason email is so popular). Is there such a thing for spreading content semi-automatically?

I'd expect either HTML or Markdown as a basis, and wonder how that would play with embeds and custom tags.

Collapse
 
ben profile image
Ben Halpern

Yeah, this is why off-the-shelf tooling might need to be built on. We took a loose standard (markdown) and built our own custom integrations (using liquid syntax), which is similar to other implementations so not entirely off-standard but still its own thing. The embeds are also fairly tightly tied to their own data and context, (like the authenticated GitHub API for example).

So there are some questions to be figured out. I think our parsers will have to spit out different versions of posts for different contexts (our embeds already don't play wonderfully with RSS readers so we could probably already be doing this to be honest), but ultimately I think we have the tools to build a lot of good shared spaces where reading config can be ported but control of the various shared spaces is federated.

Ultimately, enabling a lot of "medium-sized" web experiences to work effectively with the small things is a more fruitful path than dreaming of a web where everything is a tiny website. Medium-sized platforms and ecosystems strike me as the goldilox zone where you can put a lot of work into custom community-centric experiences without becoming overly powerful.

I think there was an era on the web where interest-based forums had a bigger and better role. I think the form-factor itself loses out to the more flexible social media idea and that has contributed to the downfall of these forums, but if we could return to some of what made those feel so intimate and awesome, we're in a good place.

(I think I went in a major tangent to reply, as I'm one to do sometimes) 😄

Collapse
 
ohadpr profile image
ohadpr

We’re excited to be a part of this initiative together with the DEV Community looking at new ways to publish, syndicate and own content on the web

Collapse
 
vuild profile image
Vuild

Ben jamming hard. 👍

Dev.to leadership team is tuned. 🔥

The web was more fun with different things everywhere.

Video is monopolized (someone?). 👀

Collapse
 
christopherkade profile image
Christopher Kade

Man, these metaphors are on point

Collapse
 
vuild profile image
Vuild

They are indeed on point Christopher. 👍

thewayoftheweb.net/should-you-stop...

@ben If you made it up, or learned about it from research = 🔥

web.archive.org/web/20081118193815...
web.archive.org/web/20081118200905... @jess @peter @maestromac 👀😳🤣

Collapse
 
ben profile image
Ben Halpern

I came to it on my own but I’m not surprised it’s also already a thing

Thread Thread
 
vuild profile image
Vuild • Edited

Nice work. Usually large site owners want to avoid the topic.

It was starting to be a thing. I wrote a bunch about this stuff on some very public tech sites (we revisit later). They managed to quash dissenting voices & replace with shills/one voice.

It's the first time I have seen someone pick it back up in a while, but watch out as it is at the core of their strategy & they don't like it.

It's a moral imperative now & the kind of thing that can actually make a real difference.

Collapse
 
taillogs profile image
Ryland G

Just the direction things should be going. Decentralized but connected.

I'm still worried that maintaining consistency and quality at scale will be the biggest obstacle (I know this is the intention behind the project, but still). I know it may be a touchy subject, but have you reached out to Free Code Camp? It seems there may be some serious idealogical overlap.

Really cool read overall!

Collapse
 
ben profile image
Ben Halpern

I know it may be a touchy subject, but have you reached out to Free Code Camp? It seems there may be some serious idealogical overlap.

I certainly would think so! We should reach out just to level set in general. We'll see how that goes. We certainly don't feel like the issues have been resolved at all but hopefully we could create a dialog where we're not speaking past each other.

maintaining consistency and quality at scale will be the biggest obstacle

Definitely. If we wanted to do something easy this probably wouldn't be what we put our effort into 😄

I'd say the big bet here is that open source code gives us eyeballs efficiencies we could never achieve on our own and creating systems which are generally big enough independently to have dedicated maintenance will be important. I think a medium number of medium-sized things generally would be better than a small number of huge things or a huge number of small things.

We shall see.

Collapse
 
andypiper profile image
Andy Piper

If you're interested in the genesis of the concept of "data gravity", check out this 2010 blog post from Dave McCrory, generally credited with coining the term :-)

Collapse
 
marcellothearcane profile image
marcellothearcane

Minor gripe - mixed metaphors are specifically things like 'hit the nail right between the nose' and 'not the sharpest cookie in the jar', which I don't think you've used.

I could only see vanilla flavoured metaphors (see what I did there?)

Collapse
 
stegriff profile image
Ste Griffiths

Something you've done much better than Mastodon, Diaspora, etc. in my opinion is to build the shining city on the hill first, get a lot of stake from users, and then generalise it and give it to people to equip the wider community.

It could have been much harder (and less successful) to try and drop that software a year ago if that had been the priority. But now you have a crowd really excited about the whole thing.

P.s. Happy to see you folks explicitly using the POSSE term (first time I've noticed it here) as it's been clear for a while that that's what you're going for.

Loving everything about this. Thanks Dev team!

Collapse
 
arcticspacefox profile image
ArcticSpaceFox

Great article! I Love DEV. For small starters like me this had a Lot of helpful Tips and conventions. Keep it up and Ben actually Made me Lauch 😂