Happy and Productive Engineers with Backstage

The Need for Internal Delivery Portals

Internal Delivery Portals (IDPs) are all the rave now. (other folks call them Internal Developer Portals but I protest - they aren't only about developers!)

And it's understandable.
We write more and more software, our systems become more modular and complex (did somebody say "microservices"?), our teams are autonomous and self-contained.
But at the end of the day we all need to join our efforts and that's where the story breaks up. The knowledge and communication is simultaneously everywhere and nowhere - Jira tickets, Confluence articles, CI/CD systems, Git repos, Slack channels, Notion spaces, Google docs, Artifact registries, you name it...

What if instead we could have a single website/app where we could always find all the relevant information and tools in order to work efficiently?

We've had this dream for a long time, but now more and more orgs are starting to make it a reality.

If you want a better understanding of the need for these portals and their basic requirements - read here.

There's an OSS for it!

There are already a number of vendors providing IDP solutions, but we know that our industry is mainly fueled by community efforts. It's no different with IDPs - there's a great OSS project that has become the de-facto industry standard for creating a portal! I'm talking about Backstage - the framework for building delivery (developer) portals. It was created at Spotify to improve the collaboration inside the engineering department and then open-sourced - so now we all can benefit from it!

According to Spotify the effects of using Backstage were:

• A 55% decrease in onboarding time for their engineers. Onboarding time was measured by time until the 10th pull request.

• 280 engineering teams inside Spotify used Backstage.

• 2,000+ backend services, 300+ websites, 4,000+ data pipelines, and 200+ mobile features onboarded to Backstage.

(Taken from this paywalled post in Pragmatic Engineer newsletter)

So yes - building a delivery portal definitely pays off.

The added benefit of Backstage is that it's a flexible, extensible system - one can pretty easily develop custom integrations on top of it.

There's a growing number of community plugins available here:

https://backstage.io/plugins/

Backstage in the Wild - the Challenges

In the last year we've helped a number of our customers to implement an IDP based on Backstage. Getting started was definitely challenging!
Here are some of the pitfalls you should be aware of if you are looking at using Backstage for your IDP effort:

Check your Reaction

First of all - the whole app is Reactjs + Nodejs written in Typescript. Not exactly your DevOps professional toolbox of choice. We're much more proficient with Python, Go, Ruby or even Java.
In fact the language itself is less of a problem, but Reactjs is a whole new world to get into.

Documentation is Lacking

Backstage devs and community invested a lot in the docs. They are beautifully written. But still - they mostly cover the basic happy path. Once you go for something more adventurous - it's very hard to understand what's not working. A few times we found ourselves reaching out to Backstage devs on Discord in order to find that hidden config field that was giving us trouble.

Plugins aren't really Plug-n-Play

Backstage is based on the concept of plugins. Which is excellent! Trouble is - these aren't the plugins we're used to from systems like Jenkins or Wordpress. In order to integrate a plugin one needs to:

• change the core code of Backstage
• rebuild the whole codebase and
• redeploy their IDP

Not exactly the perfect operational flow for a live system that can potentially change multiple times a month. Especially weird now - that we've learned the pains of monolithic deployments and how to mitigate them with modularity.

Upgrades are Scary

So - as we said - configuring plugins and basically adding any other changes to Backstage involves code modifications. This effectively turns your Backstage installation into a fork of the original codebase. Which makes every upgrade a code merge that's on you to perform. And code merges are ... scary.
Backstage developers even created a tool to generate code diffs that one needs to manually integrate in their installation in order to upgrade. Which we're very thankful for. Makes it easier. But still scary.

Getting Everyone On Stage

After you've muscled through all the technical challenges - you hit the hardest part - organizational adoption. At the core of every IDP sits the software catalog - the registry of all the components developed and used at your organization. So for an effective adoption you need:

1. Collect all that data into Backstage (converting it to Backstage-compatible format)
2. Persuade all the engineers to consume that data from the IDP and not from wherever they found the data prior to that. (Old habits die hard)
3. Continuously update the data - so it stays relevant and useful.

For a large org with hundreds of devs and dozens of software components this is no easy feat. More often than not - adoption fails or stumbles. Mainly because there's no true DX (developer experience) expertise in the organization. Backstage itself is just a framework - turning it into a usable and valuable product becomes the job of the team responsible for its rollout. This requires a very specific skillset which is lacking in most places.

So if you've been considering rolling out Backstage at your org - think again... Or -

Use a Backstage Distro

In order to make Backstage adoption and day-to-day operation easier and more valuable for our customers we've been working on our own Backstage distro built to run on Kubernetes.

The goal is to provide an enterprise ready product with:

• Seamless deployment experience
• Automated Catalog Discovery
• Plugin management without rebuilds (for major plugins)
• Manageable RBAC and enterprise-level auditing out of the box
• Integrated Observability with customizable metrics.
• Optional reporting and Finops extensions (once you have everyone in the org in one place, cataloged and documented - makes a lot of sense to generate reports and manage costs accordingly)

Sounds interesting? Feel free to signup at our website to get notified when we release a public beta.

And if the topic of IDPs and Backstage excites you - subscribe to my posts. Here are some of the topics I'm planning to cover:

• Reviews of proprietary IDP offerings (Port, Cortex, OpsLevel, etc)
• How to join Backstage OSS community and contribute to Backstage OSS
• Technical posts on how to do things in Backstage
• Introduction to Platform Design Principles

I'm also doing a Backstage workshop at Cloud Native Day Switzerland in September, so if you're around - do sign up!

Comments

[Addison]

Super nice article - I'm curious, is there a reason you'd prefer to go an OSS route vs using something that has a lot more directly out of the box?

I can see the benefits of having more control over your IDP - but also requires a lot more effort to maintain/setup

[Ant(on) Weiss]

thanks for the feedback!
Regarding your question - well, you just laid it out there. We think using OSS hits the sweet spot right in the middle of the build vs. buy dichotomy.

• you're not locked in to a proprietary solution
• you enjoy community-supported standards
• you retain most of flexibility and control

Yes, it is more effort - but it all pays off in the long run.
Platform engineering is a long game.