Notes from BackstageCon 2022

All talks available on this Youtube Playlist.

• Hosts - Suzanne Daniels - Spotify, DevRel, Martina Iglesias Fernandez - Roadie, CTO

Opening Remarks - Suzanne Daniels

• Happy Developers Make Happy Code
• Devs are spending less than 30% of their time writing code
• Backstage Stats
  • 1,500+ Contributors
  • 3,200+ Project Forks
  • 400+ Adopters
  • 7,300+ Discord Members
  • 13,000+ contributions
• Development started at Spotify in 2014, open source since 2020
  • Recently moved to CNCF Incubator status
  • 0 released
  • Passed security audit
  • https://backstage.io/blog/2022/08/23/backstage-security-audit

10x the Effectiveness of Dev Teams with Backstage - David Tuite, CEO, Roadie

• Last 15 yrs brought great strides in eng effectiveness
  • Microservices
  • CI/CD
  • IDEs
• Cost is fragmentation
• Next 15 yrs about defragmentation
• Great code only matters if people knows it exists
• Backstage allows engineers to know about the great code, and use it instead of building new code from scratch repeatedly
• Templates allow teams to get started with new things quickly
• Need to organize with other teams like Sec and Infra, Backstage enables this

Building for Developer Happiness - Dave Zolotusky, Principal Engineer, Spotify

• Devs are essentials
• Every business is now a software business
• Rising complexity to develop and scale
  • DevSecFinOps
• Devs at Spotify tell us that they're happy in their jobs, and they stay longer than the industry average
• Effectiveness = f(Productivity, Happiness)
  • Unhappy Devs don't do their best work, and deliver bad code
  • 
• Developers should be able to choose the right tool for the right job, and Backstage can help nudge to use the right tools, and reduce friction
• Goal to make it a critical part of every devs daily workflow
• Spotify wants to continue to share how they
• Spotify Plugins for Backstage
  • Available for all adopters
  • Open Sourcing homegrown and newly built plugins
  • Freemium model
  • This will be a paid solution
  • Highlights list
  • Soundcheck
    • Ensure quality, reliability and alignment of software architecture throughout your org.
  • RBAC
    • Use roles to control access to actions and data in Backstage
  • Skill Exchange
    • Build a marketplace for short-term learning and growth opportunities in R&D
  • Pulse
    • Track metrics that matter and identify gaps in your engineering practices and tools.
  • Insights
    • Measure usage of Backstage in your organization
• 

Providing a Curated Experience for Developers with Backstage - Valentina Alaria, Sr. Director, VMware Tanzu

• Why do we care about Developer Experience?
  • Business depend on DevEx, they depend of developer productivity
• Orgs that have higher dev productivity are fundamentally more effective
  • Able to support bigger and greater innovation
  • Able to react to situations, more effectively
• Enterprise Golden Path
  • https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem/
• Curated yet Customizable is key to offering order, best practices, and predefined solutions to help devs be consistent, and focus on new functionality vs frequent rebuilding and hunting for tools
• Paraphrase - "All about being able to enable security end to end, while reducing burden on developers"
  • It means that you can support various LOB (Lines of Business) needs without having to rebuild every workflow for each team
• How can we enable coordination between teams, especially dev and ops
• VMWare found themselves at a fork in the road, between an in house solution or an integrated version of Backstage
• Initial Backstage releases didn't fit their needs, but development on the product rapidly matured, and so they joined the community
• Backstage now powers VMWare Tanzu Application Platform

The Evolution of Backstage Backends - Patrik Oldsberg & Johan Haals, Spotify

• Patrick
• Johan - sick, replacement Charles
• Focus on how Backstage Backends, is changing
• During open sourcing, switch to all Node.JS
• No initial backend plugins
• Goal to enable elaborate plugins, but provide consistent patterns of development
  • Simpler Installation
  • Fewer Breaking Changes
  • Less Code
  • Recognizable
  • Easy to evolve and extend
• Current: lots of copy and paste
• New system enables lightweight customizations, while having baked in templating and access for common usage patterns
• 
• New model allows sharing of "factories" that can be customized but by default allow it to work between different systems without modification
• This new model will require a migration
• New plugin platform is available now https://backstage.io/link?new-backend
• No specific timeline but hoping to launch early next year

Building an Internal Developer Platform with Backstage Plugins - Taras Mankovski, The Frontside Software, Inc

• Era of DevX - Proactive Investment in DX
  • Even 5 years ago, there was little enthusiasm for improving DevX
• DX Leads to value
• How do we continue to develop the community around backstage
• Leads DX Consulting company - Frontside
  • Backstage Professional Services Partner
• New Plugin Demo
• Internal Developer Platform Tools
  • Allows dev to download a tool that allows them to use a Backstage Instance via CLI
  • Customized for specific instance to enable seamless login and usage
  • 
• Live demo
• Backstage App is not an IDE
• Provides limited use cases
  • Discovery
  • Templating
  • Doesn't support getting runtime information from other sources
  • Outside of the existing Kubernetes integration
  • Cannot manage secrets
  • Need to manually control them via different systems and tools
  • Cannot manage releases
• Dream tool, allows integrated flows between Backstage and IDEs
  • Give developers the option to choose the right tool, to help eliminate noise, and reduce friction
• "Internal Developer Platforms are a combination of all tools that developers use to run their software."
• Future Ideal State
  • Environments exist in Backstage
  • Different cards following existing UI, showing which deployments exist for a specific user/team
  • Easily see how it connects to Git and various levels (prod/dev/staging)
  • 
  • Secrets Management
  • Releases
  • Logs
• Goal: How to make a Heroku-like DX on IDPs
• 
• Code for IDP will be available tomorrow
  • https://frontside.com/backstage
• 

Enabling Developer Experience Journey in U.S. Bank - Ravikumar Tadikonda & Poonam Garg, U.S. Bank

• Lead DevX and DevOps Platform at US Bank
• 5th largest Bank in the US, operate globally
• Mission: make it easier to create and manage modern best-of-class software products
• What should we tackle first
  • Our developer and engineer experience, as they constantly faced challenges in building and maintaining solutions
• Common Engineering Challenges
  • 
• Attempted to build a custom inhouse solution
• After that didn't succeed, they retro'd what they could improve and how that could be possible
  • Key: provide a single pane of glass experience
• Comprehensive Approach:
  • How do we enable devs to create and manage
  • Discover and Reuse
  • Insight and Control
• Concerns that the environment is highly regulated industry
• Created user personas
  • Engineers
  • Auditors
  • Managers
  • Compliance and risk control owners
  • Building a platform that can be extended to support future roles
• How is Backstage part of our journey?
  • Focus on Create and Manage at this stage
  • Blueprints/templates
• High level architecture
  • 
• Q&A
  • Rec: Karl Haworth - American Airlines Backstage Implementation - https://www.youtube.com/watch?v=0hdWil6cXw0
  • Manages 1000s of internal applications and even more sub services
  • Have to use approved Open Source libraries and versions
  • Cataloging is key to successful management at US Bank
  • How do you manage catalogue accuracy while aggregating multiple catalogs?
  • System is not 100% comprehensive
  • Use entity providers to sync asynchronous
    • Backstage first then down to custom tools
    • Vice versa for other tools
  • Longterm goals to just be Backstage
  • How did you get started with Backstage?
  • Took one demo java app and it took about a month to create the first template from that app

Leading Architectural Change with Backstage - Andy Hoffman, Caribou

• DevOps manager at Caribou, FinTech
• Caribou's Story
  • Main concern 2 years ago was Cashflow
  • Originally named MotoReFi
  • Ruby Monolith
  • 10X Growth during the Pandemic thru July 22
  • Need to step back and decompose complexity growth
  • Moved to K8s + ArgoCD
  • Where to go from here?
  • Regulated industry
  • Culture is metrics driven, experimental, and psych safety focused
• Understanding Obstacles
  • "It's impossible to guarantee any engineer you hire will understand or have experience with the full tech stack at your company."
  • Custom Shop to Assembly Line
• How Dev Expectations have changed
  • This shift has happened in the last 3 years very quickly
  • Must have friction remove
  • Need to develop to a polished level internal applications
• K8s is getting a bad rep
  • Complexity and overhead isn't worth it from devs
  • K8s is making the apps less stable
  • New Services take too long to stand up
  • Copypasta everywhere and too much YAML
  • 
• Naming is Hard
  • As apps become distributed, how do you name your functions and apps
• Infra and Sec teams are slowing us down
  • K8s minimal sec footprint can hinder wide open dev practices
• Anti Pattern of Distributed Monoliths is arising
  • All components are needed for any single component to work
• Cloud Native Platforms around Backstage
  • Backstage can reduce complexity
  • DevOps used to enable knowledge sharing
  • Backstage eliminates the need for DevOps/Infra/Automation team in a lot of cases
• ANTLERS - Automatic Normalized Toolchain for Launching Easily Releasable Systems
• Didn't have room to internally scale
  • Hired Roadie to build out their Backstage and ANTLERS
  • Call the system the Barn
• Architecture
  • Hard to know where you'll end up, but it's easy to know where you will start
• Simple & Intuitive Architecture == Simple & Intuitive Scaffolder Implementation
• Define Scope and Boundaries early
  • Be intentional about onboarding as it will lead to positive experiences
  • Rocky onboarding will leave a bad taste
• Dove into Argo
  • Provisions all of the infra as part of the ArgoCD implementation
• World of Pain is when a DevOps engineer doesn't think through about conventions when templating YAML
• Registering Resources at the Domain Level, sign off at levels driven by eng org
• Git driven governance
• Leverage Helm + Backstage Scaffolder
• Maturity Model -> Focus on Lifecycle Management including building in time for known upgrades as systems become routine
• "How do we ask other teams to accept change, without acknowledging their suffering"

What is This Tech Insights Anyways? - Jussi Hallila & Irma Solakovic, Roadie

• Tech Insights Plugin
• 2 Core Engineers who developed it
• Searching for an automated way to keep things organized and understand the relative value and usage (software quality)
• Introduced Scorecards to measure the quality of software
  • 
• Focused on visualizing what matters most
  • EX Snyk critical vulns over time
• Identify teams that need the most support
  • Low quality ratings across related apps visualizes a previously hidde struggle
• Automated nudges via Slack/ChatOps
• https://roadie.io/backstage/plugins/tech-insights/
• Backstage directly supports SQLite and Postgres
• Facts
  • Data retrieved from external sources
  • Data already in Backstage
  • Can be customized to query against any database but designed around the SQLite or Postgres
• Calculations
  • Checks - Boolean/True or False
  • Aggegations, Accumulations and Indicators
  • Visualize data from databases and retrieved facts
• Visualizations
  • Scorecards
  • Graph Cards
  • Aggregate visualizations
  • Others
  • Initiatives, migrations and other tech insights
    • Ex: Tracking how teams are doing on migrations like Helm2 to Helm3
    • They used Python2 to Python3
    • Link: How to get an overview of your tech health (Tech Insights plugin) in Backstage (Demo)
• How to Use Tech Insights now
  • Create fact retrievers
  • Create checks
  • Configure the app to display scorecards based on checks
• Open to contributions?
• Available on Hosted Backstage at Roadie.com

Measuring Developer Happiness: Analytics for Backstage - Eric Peterson, Spotify

• @iameap
• Worked at Tableau for 8 years, been doing Backstage for 2 years, lives in Rural Sweden
• DevX Story
  • The Review, Build, Feedback Loop
  • Time spent waiting for either peer or CI/CD feedback between builds
• Translating the DX into Data in Backstage
• Show the grammar of Backstage so you can visualize your own devs stories
• Backstage Analytics Architecture
  • 
  • Subject -> Noun
  • Action -> Verb
  • Attributes -> Adjectives
  • Context -> Adverbs
  • Extension-Derived Context
• Works out of the box, if you use the core createPlugin function in your plugins

How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.

• 200K objects needed to be ingested from a single provider
• How do you make data objects you have no control over, searchable?
• HP has just about everything new and everything old in terms of tech stack
• Backstage kept getting slower as their usage increased
• V1 - mostly out of the box Backstage:
  • Catalog Processors emitting locations, read by other Catalog Processors
  • Recursion!
  • No control or schedule around ingestion
  • Locations polluting the catalog
  • Entities would eventually show up, but took a long time, and wasn't reliable
• V2 - Backstage with Entity Providers
  • Still lots of overlap and recursion
  • Didn't resolve slowness, and may have made it worse
  • Did have benefits in ability to clear orphaned resources automatically
  • Esp users who are no longer active
• 
• Both design patterns fail to meet scaling needs out of the box
• Started running into DB Concurrency issues with V2
• Solution
  • Incremental Entity Providers
  • Built in incremental backoff
  • Data Sources must support pagination
  • Optimizing the processing loops
  • Commiting to front loading all async operations into the Incremental Entity Provider, and banned them from Catalog Processors
• Success in decreasing the the running average from 4+ hours in mid July to about a half hour for an entity to be found and scheduled, to then be ingested
  • Longest they ever saw was over 1 day
• Newest iteration in dev is ingesting new entities in about 15 minutes
  • With a full challenge of ingesting all resources from zero!
• Looking to open sourcing it in November of this year
• How can the Backstage community focus not just on making tools that get new teams from 0 to 1 quickest, but solutions that will work in 2+ years
  • Enterprise functional deployments
• 

A Picture is Worth a Thousand Words: Bring Data Visualization in Backstage - Olivier Liechti, Avalia Systems

• Deep dive demo into visualizing data with Vega.JS and D3.JS and how this can be used inside of Backstage
• Similar to Tech Insights, but need deep understanding of D3.JS and Vega.JS, if skill is already there should be seamless

What We Learned During Our Journey in Using Backstage at DAZN - Marco Crivellaro, DAZN

• DAZN - Italian Media Company
• Dev Success provides support to 700+ devs in their org
• 
• The dazn-manifest
• Started to integrate with Backstage in early 2021
• Start small and build piece by piece
  • Can get way too deep in the plugin ecosystem
  • Get to the core of your needs, and get 1 thing working, and then the next
  • Don't try and launch every plugin, as devs will dive deeper than you
• They focused on Software Catalog -> Tech Docs and are now working on Scaffolder
• Don't be afraid to get your hands dirty
• Could be painful though
  • Custom integrations can be a rabbit hole
• Don't let Backstage become your single source of truth
  • Hinderance to disaster recovery if you lose backstage

Lightning Talks

Talking in Backstage: How to Navigate the Business and Engineering Vocabulary - Jorge Lainfiesta, Roadie

• Why? Language is hard
• Developed Course: Introduction to Backstage - https://www.classcentral.com/course/introduction-to-backstage-developer-portals-made--110320
• Adopting Backstage is an org-encompassing initiative
  • Creates a lot of possibility, but will need to work through a lot of org wide friction
• Terms
  • Doers - Get the work of backstage running
  • Sponsors - Communicate and vouch for the need during the building phase
  • Users - Interact with backstage and provide a reason for it to exist
  • Backstage is a platform for making developer portals
  • Developer Portal - A self-service one-stop shop
  • Discoverability - How easy it is to find something
  • Software Catalog - a structured collection of all software assets
  • Not just a list!
  • Can have 100s of 1000s of entities that are searchable and consumable in Backstage
  • Entities - a unit of information stored in a catalog
  • Search - a way of discovering across sources
  • Plugins - your Backstage features
  • New features are added by building internal or integrating external plugins
  • Self service - allows you to access or build something without needing to start a communication process
  • Open Source - Not Free
  • Community - Power

Building and Contributing Our First Plug-In at TELUS - Isaiah Thiessen, TELUS

• Over 80 open source plugins in Backstage since launch ~ 2 years ago
• How he developer his first plugin
• Creator of the Backstage Dynatrace plugin
• Fell in love with how good the Backstage plugin docs were
• Backstage Storybook - https://backstage.io/storybook/?path=/story/plugins-examples--plugin-with-data
• Learned how to use React as a backend dev
• Combined Frontend Components -> Backstage Backend Proxy -> API
• Worked with Tellus' Open Source office to get it shipped