DEV Community: Camille Clayton

Notes from BackstageCon 2022

Camille Clayton — Thu, 03 Nov 2022 15:58:18 +0000

All talks available on this Youtube Playlist.

Hosts - Suzanne Daniels - Spotify, DevRel, Martina Iglesias Fernandez - Roadie, CTO

Opening Remarks - Suzanne Daniels

Happy Developers Make Happy Code
Devs are spending less than 30% of their time writing code
Backstage Stats
- 1,500+ Contributors
- 3,200+ Project Forks
- 400+ Adopters
- 7,300+ Discord Members
- 13,000+ contributions
Development started at Spotify in 2014, open source since 2020
- Recently moved to CNCF Incubator status
- 0 released
- Passed security audit
- https://backstage.io/blog/2022/08/23/backstage-security-audit

10x the Effectiveness of Dev Teams with Backstage - David Tuite, CEO, Roadie

Last 15 yrs brought great strides in eng effectiveness
- Microservices
- CI/CD
- IDEs
Cost is fragmentation
Next 15 yrs about defragmentation
Great code only matters if people knows it exists
Backstage allows engineers to know about the great code, and use it instead of building new code from scratch repeatedly
Templates allow teams to get started with new things quickly
Need to organize with other teams like Sec and Infra, Backstage enables this

Building for Developer Happiness - Dave Zolotusky, Principal Engineer, Spotify

Devs are essentials
Every business is now a software business
Rising complexity to develop and scale
- DevSecFinOps
Devs at Spotify tell us that they're happy in their jobs, and they stay longer than the industry average
Effectiveness = f(Productivity, Happiness)
- Unhappy Devs don't do their best work, and deliver bad code
Developers should be able to choose the right tool for the right job, and Backstage can help nudge to use the right tools, and reduce friction
Goal to make it a critical part of every devs daily workflow
Spotify wants to continue to share how they
Spotify Plugins for Backstage
- Available for all adopters
- Open Sourcing homegrown and newly built plugins
- Freemium model
- This will be a paid solution
- Highlights list
- Soundcheck
  - Ensure quality, reliability and alignment of software architecture throughout your org.
- RBAC
  - Use roles to control access to actions and data in Backstage
- Skill Exchange
  - Build a marketplace for short-term learning and growth opportunities in R&D
- Pulse
  - Track metrics that matter and identify gaps in your engineering practices and tools.
- Insights
  - Measure usage of Backstage in your organization

Providing a Curated Experience for Developers with Backstage - Valentina Alaria, Sr. Director, VMware Tanzu

Why do we care about Developer Experience?
- Business depend on DevEx, they depend of developer productivity
Orgs that have higher dev productivity are fundamentally more effective
- Able to support bigger and greater innovation
- Able to react to situations, more effectively
Enterprise Golden Path
- https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem/
Curated yet Customizable is key to offering order, best practices, and predefined solutions to help devs be consistent, and focus on new functionality vs frequent rebuilding and hunting for tools
Paraphrase - "All about being able to enable security end to end, while reducing burden on developers"
- It means that you can support various LOB (Lines of Business) needs without having to rebuild every workflow for each team
How can we enable coordination between teams, especially dev and ops
VMWare found themselves at a fork in the road, between an in house solution or an integrated version of Backstage
Initial Backstage releases didn't fit their needs, but development on the product rapidly matured, and so they joined the community
Backstage now powers VMWare Tanzu Application Platform

The Evolution of Backstage Backends - Patrik Oldsberg & Johan Haals, Spotify

Patrick
Johan - sick, replacement Charles
Focus on how Backstage Backends, is changing
During open sourcing, switch to all Node.JS
No initial backend plugins
Goal to enable elaborate plugins, but provide consistent patterns of development
- Simpler Installation
- Fewer Breaking Changes
- Less Code
- Recognizable
- Easy to evolve and extend
Current: lots of copy and paste
New system enables lightweight customizations, while having baked in templating and access for common usage patterns
New model allows sharing of "factories" that can be customized but by default allow it to work between different systems without modification
This new model will require a migration
New plugin platform is available now https://backstage.io/link?new-backend
No specific timeline but hoping to launch early next year

Building an Internal Developer Platform with Backstage Plugins - Taras Mankovski, The Frontside Software, Inc

Era of DevX - Proactive Investment in DX
- Even 5 years ago, there was little enthusiasm for improving DevX
DX Leads to value
How do we continue to develop the community around backstage
Leads DX Consulting company - Frontside
- Backstage Professional Services Partner
New Plugin Demo
Internal Developer Platform Tools
- Allows dev to download a tool that allows them to use a Backstage Instance via CLI
- Customized for specific instance to enable seamless login and usage
Live demo
Backstage App is not an IDE
Provides limited use cases
- Discovery
- Templating
- Doesn't support getting runtime information from other sources
- Outside of the existing Kubernetes integration
- Cannot manage secrets
- Need to manually control them via different systems and tools
- Cannot manage releases
Dream tool, allows integrated flows between Backstage and IDEs
- Give developers the option to choose the right tool, to help eliminate noise, and reduce friction
"Internal Developer Platforms are a combination of all tools that developers use to run their software."
Future Ideal State
- Environments exist in Backstage
- Different cards following existing UI, showing which deployments exist for a specific user/team
- Easily see how it connects to Git and various levels (prod/dev/staging)
- Secrets Management
- Releases
- Logs
Goal: How to make a Heroku-like DX on IDPs
Code for IDP will be available tomorrow
- https://frontside.com/backstage

Enabling Developer Experience Journey in U.S. Bank - Ravikumar Tadikonda & Poonam Garg, U.S. Bank

Lead DevX and DevOps Platform at US Bank
5th largest Bank in the US, operate globally
Mission: make it easier to create and manage modern best-of-class software products
What should we tackle first
- Our developer and engineer experience, as they constantly faced challenges in building and maintaining solutions
Common Engineering Challenges
Attempted to build a custom inhouse solution
After that didn't succeed, they retro'd what they could improve and how that could be possible
- Key: provide a single pane of glass experience
Comprehensive Approach:
- How do we enable devs to create and manage
- Discover and Reuse
- Insight and Control
Concerns that the environment is highly regulated industry
Created user personas
- Engineers
- Auditors
- Managers
- Compliance and risk control owners
- Building a platform that can be extended to support future roles
How is Backstage part of our journey?
- Focus on Create and Manage at this stage
- Blueprints/templates
High level architecture
Q&A
- Rec: Karl Haworth - American Airlines Backstage Implementation - https://www.youtube.com/watch?v=0hdWil6cXw0
- Manages 1000s of internal applications and even more sub services
- Have to use approved Open Source libraries and versions
- Cataloging is key to successful management at US Bank
- How do you manage catalogue accuracy while aggregating multiple catalogs?
- System is not 100% comprehensive
- Use entity providers to sync asynchronous
  - Backstage first then down to custom tools
  - Vice versa for other tools
- Longterm goals to just be Backstage
- How did you get started with Backstage?
- Took one demo java app and it took about a month to create the first template from that app

Leading Architectural Change with Backstage - Andy Hoffman, Caribou

DevOps manager at Caribou, FinTech
Caribou's Story
- Main concern 2 years ago was Cashflow
- Originally named MotoReFi
- Ruby Monolith
- 10X Growth during the Pandemic thru July 22
- Need to step back and decompose complexity growth
- Moved to K8s + ArgoCD
- Where to go from here?
- Regulated industry
- Culture is metrics driven, experimental, and psych safety focused
Understanding Obstacles
- "It's impossible to guarantee any engineer you hire will understand or have experience with the full tech stack at your company."
- Custom Shop to Assembly Line
How Dev Expectations have changed
- This shift has happened in the last 3 years very quickly
- Must have friction remove
- Need to develop to a polished level internal applications
K8s is getting a bad rep
- Complexity and overhead isn't worth it from devs
- K8s is making the apps less stable
- New Services take too long to stand up
- Copypasta everywhere and too much YAML
Naming is Hard
- As apps become distributed, how do you name your functions and apps
Infra and Sec teams are slowing us down
- K8s minimal sec footprint can hinder wide open dev practices
Anti Pattern of Distributed Monoliths is arising
- All components are needed for any single component to work
Cloud Native Platforms around Backstage
- Backstage can reduce complexity
- DevOps used to enable knowledge sharing
- Backstage eliminates the need for DevOps/Infra/Automation team in a lot of cases
ANTLERS - Automatic Normalized Toolchain for Launching Easily Releasable Systems
Didn't have room to internally scale
- Hired Roadie to build out their Backstage and ANTLERS
- Call the system the Barn
Architecture
- Hard to know where you'll end up, but it's easy to know where you will start
Simple & Intuitive Architecture == Simple & Intuitive Scaffolder Implementation
Define Scope and Boundaries early
- Be intentional about onboarding as it will lead to positive experiences
- Rocky onboarding will leave a bad taste
Dove into Argo
- Provisions all of the infra as part of the ArgoCD implementation
World of Pain is when a DevOps engineer doesn't think through about conventions when templating YAML
Registering Resources at the Domain Level, sign off at levels driven by eng org
Git driven governance
Leverage Helm + Backstage Scaffolder
Maturity Model -> Focus on Lifecycle Management including building in time for known upgrades as systems become routine
"How do we ask other teams to accept change, without acknowledging their suffering"

What is This Tech Insights Anyways? - Jussi Hallila & Irma Solakovic, Roadie

Tech Insights Plugin
2 Core Engineers who developed it
Searching for an automated way to keep things organized and understand the relative value and usage (software quality)
Introduced Scorecards to measure the quality of software
Focused on visualizing what matters most
- EX Snyk critical vulns over time
Identify teams that need the most support
- Low quality ratings across related apps visualizes a previously hidde struggle
Automated nudges via Slack/ChatOps
https://roadie.io/backstage/plugins/tech-insights/
Backstage directly supports SQLite and Postgres
Facts
- Data retrieved from external sources
- Data already in Backstage
- Can be customized to query against any database but designed around the SQLite or Postgres
Calculations
- Checks - Boolean/True or False
- Aggegations, Accumulations and Indicators
- Visualize data from databases and retrieved facts
Visualizations
- Scorecards
- Graph Cards
- Aggregate visualizations
- Others
- Initiatives, migrations and other tech insights
  - Ex: Tracking how teams are doing on migrations like Helm2 to Helm3
  - They used Python2 to Python3
  - Link: How to get an overview of your tech health (Tech Insights plugin) in Backstage (Demo)
How to Use Tech Insights now
- Create fact retrievers
- Create checks
- Configure the app to display scorecards based on checks
Open to contributions?
Available on Hosted Backstage at Roadie.com

Measuring Developer Happiness: Analytics for Backstage - Eric Peterson, Spotify

@iameap
Worked at Tableau for 8 years, been doing Backstage for 2 years, lives in Rural Sweden
DevX Story
- The Review, Build, Feedback Loop
- Time spent waiting for either peer or CI/CD feedback between builds
Translating the DX into Data in Backstage
Show the grammar of Backstage so you can visualize your own devs stories
Backstage Analytics Architecture
- Subject -> Noun
- Action -> Verb
- Attributes -> Adjectives
- Context -> Adverbs
- Extension-Derived Context
Works out of the box, if you use the core createPlugin function in your plugins

How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.

200K objects needed to be ingested from a single provider
How do you make data objects you have no control over, searchable?
HP has just about everything new and everything old in terms of tech stack
Backstage kept getting slower as their usage increased
V1 - mostly out of the box Backstage:
- Catalog Processors emitting locations, read by other Catalog Processors
- Recursion!
- No control or schedule around ingestion
- Locations polluting the catalog
- Entities would eventually show up, but took a long time, and wasn't reliable
V2 - Backstage with Entity Providers
- Still lots of overlap and recursion
- Didn't resolve slowness, and may have made it worse
- Did have benefits in ability to clear orphaned resources automatically
- Esp users who are no longer active
Both design patterns fail to meet scaling needs out of the box
Started running into DB Concurrency issues with V2
Solution
- Incremental Entity Providers
- Built in incremental backoff
- Data Sources must support pagination
- Optimizing the processing loops
- Commiting to front loading all async operations into the Incremental Entity Provider, and banned them from Catalog Processors
Success in decreasing the the running average from 4+ hours in mid July to about a half hour for an entity to be found and scheduled, to then be ingested
- Longest they ever saw was over 1 day
Newest iteration in dev is ingesting new entities in about 15 minutes
- With a full challenge of ingesting all resources from zero!
Looking to open sourcing it in November of this year
How can the Backstage community focus not just on making tools that get new teams from 0 to 1 quickest, but solutions that will work in 2+ years
- Enterprise functional deployments

A Picture is Worth a Thousand Words: Bring Data Visualization in Backstage - Olivier Liechti, Avalia Systems

Deep dive demo into visualizing data with Vega.JS and D3.JS and how this can be used inside of Backstage
Similar to Tech Insights, but need deep understanding of D3.JS and Vega.JS, if skill is already there should be seamless

What We Learned During Our Journey in Using Backstage at DAZN - Marco Crivellaro, DAZN

DAZN - Italian Media Company
Dev Success provides support to 700+ devs in their org
The dazn-manifest
Started to integrate with Backstage in early 2021
Start small and build piece by piece
- Can get way too deep in the plugin ecosystem
- Get to the core of your needs, and get 1 thing working, and then the next
- Don't try and launch every plugin, as devs will dive deeper than you
They focused on Software Catalog -> Tech Docs and are now working on Scaffolder
Don't be afraid to get your hands dirty
Could be painful though
- Custom integrations can be a rabbit hole
Don't let Backstage become your single source of truth
- Hinderance to disaster recovery if you lose backstage

Lightning Talks

Talking in Backstage: How to Navigate the Business and Engineering Vocabulary - Jorge Lainfiesta, Roadie

Why? Language is hard
Developed Course: Introduction to Backstage - https://www.classcentral.com/course/introduction-to-backstage-developer-portals-made--110320
Adopting Backstage is an org-encompassing initiative
- Creates a lot of possibility, but will need to work through a lot of org wide friction
Terms
- Doers - Get the work of backstage running
- Sponsors - Communicate and vouch for the need during the building phase
- Users - Interact with backstage and provide a reason for it to exist
- Backstage is a platform for making developer portals
- Developer Portal - A self-service one-stop shop
- Discoverability - How easy it is to find something
- Software Catalog - a structured collection of all software assets
- Not just a list!
- Can have 100s of 1000s of entities that are searchable and consumable in Backstage
- Entities - a unit of information stored in a catalog
- Search - a way of discovering across sources
- Plugins - your Backstage features
- New features are added by building internal or integrating external plugins
- Self service - allows you to access or build something without needing to start a communication process
- Open Source - Not Free
- Community - Power

Building and Contributing Our First Plug-In at TELUS - Isaiah Thiessen, TELUS

Over 80 open source plugins in Backstage since launch ~ 2 years ago
How he developer his first plugin
Creator of the Backstage Dynatrace plugin
Fell in love with how good the Backstage plugin docs were
Backstage Storybook - https://backstage.io/storybook/?path=/story/plugins-examples--plugin-with-data
Learned how to use React as a backend dev
Combined Frontend Components -> Backstage Backend Proxy -> API
Worked with Tellus' Open Source office to get it shipped

Medium Services Manifesto

Camille Clayton — Wed, 23 Feb 2022 13:23:28 +0000

This page is an ode to the middle, properly scaled, efficient, and manageable systems. It pays homage to building services that will be useful, profitable, and won't leave you crying at midnight.

Who I am: I am Camille Clayton. I have 8 years of system design and engineering experience, sprinkling in application development, user training, documentation writing, and some other twists and turns. Through this article, I hope to share a bit about building human-centered systems, question the status quo of monolith vs. microservices, and share a bit of love for well-rounded systems!

What Is Unique About a Medium System

A defining characteristic is a system with essential business functions, value, and known boundaries regarding maximum user input and interaction. It will have known and manageable external constraints. This does not apply to public user-facing apps with an unbounded maximum of users and need to scale under unexpected load; see the Slashdot effect.
A medium system may need to scale dramatically, but it’s usually during planned or known times, like financial systems during tax season.
A medium system may have once been a large-scale system, especially in government, finance, scientific research, and other early computing adopting industries. As user levels and logic needs stay relatively static, these industries may not need major system resource increases, especially for internal systems.

Know Your Environment
- If you aren't at the scale of Google or Amazon, where you're registering millions and possibly billions of transactions per second, you can likely get away with a simpler design pattern. This will save you time, energy and headaches.
- If you are near that scale or actively facing scaling problems, read Google or Amazon’s whitepapers on their design choices. There are clear reasons why these large, complex, and global scale systems exist; it will also exemplify times and opportunities to use them. Ex: CDNs (Content Delivery Networks).
Be Willing to Sacrifice
- Not every service, piece of code, or server stood up will be useful in one or more years down the road.
- Work to develop a process to sunset services, deprecate systems, and annotate documentation when these items are no longer in use.
- Keeping your system lean enables you to develop new features more easily and minimize the amount of upgrading and patching for under or unused systems.
Design the Systems You Need
- It's easy to be fascinated by new tooling, paradigms, and technologies, but the reality is most software already has a solution.
- By relying on known technologies, and ones you and your team are comfortable with, you will significantly reduce your internal friction and ramp up time. In the long term, this gives you a runway for future innovation.
- Dan McKinley talks more about relying on foundational technologies in their essay "Choose Boring Technologies"
Choose a Queueing Service
- Understand its quirks and set up alerts, and never let anyone enable a second queuing service.
You Don't Need 2 Databases
- If you're deciding on whether to expand your schema or add a second database, you likely have a different need. In that case, you should hire a DBA or a Data Consultant.
- Unless you're writing millions of transactions a second, and need the ability to have data be desynchronized to get additional speed of writes, you're better off reevaluating your current schema.
- You should look at optimization technologies like Redis, or management technologies including Flyway or Liquibase.
Automation is Essential
- The longer your team takes to decide on an automation platform, or if they split into multiple platforms, the longer each revision to infrastructure and code will take.
- Automation is a keystone for the functionality or dysfunctionality of your teams. Automation platforms should be boring foundational tools that enable you to to create higher-level products with expected repeatability quickly.
- If your tooling is not repeatable, this should be prioritized and cleared as a roadblock.
Everything You Add Will Need to Be Maintained
- Whenever a system, service, app, or site is being developed, questions need to be asked. Who will maintain this new thing? What team will it be dedicated to, and will it be shared amongst development groups? Will it be owned by a platform team?
- This should be decided early on, documented, and sent out via email with positive confirmation.
- Deciding these things after something has been implemented can devolve into months-long turf wars that no one wants or needs.
Know Your Users
- Listen to your users. Figure out what they’re using and if they have any pain points.
- Happy users will let you test new systems out and innovate accordingly. Unhappy users will keep you pinned down and reworking until it's right.
Work With Your Stakeholders
- In consulting, this is usually an external role. In medium to large companies, this is usually internal, and it goes by many names, whether it's a Product Manager, Project Manager, Application Owner, and more. Regardless, this person or persons holds the keys between your system and end users. Having this individual on your team’s side is vital to keeping focused, clear objectives and aligning your business and teams for success.
- If you're an engineer, an architect, or otherwise involved in the day-to-day of building and maintaining a system, developing a good working relationship with this person is crucial. This will allow you to get buy-in for the hard stuff, refactoring and removing tech debt, and the ability to innovate beyond changing the colors of buttons.
- Invest in your stakeholders work happiness, as they hear from management and users directly. They often know of additional aggregate pain points, even if they don't bring them up at every sprint review. Even if you can't work on them today, sorting out the longer-term issues can often help you prevent them from becoming active fires down the road. It can allow you to find ways to incrementally mitigate them if there's time between sprints.
Share Your On-Call Rotation
- Historically, on-call was the responsibility of Ops folks. In the 2010s, there have been blog posts and thinkpieces that Devs should be part of the rotation or if SREs should guide in the formation of rotations. In working through various rotations since 2014, I've come to the simpler conclusion: everyone should be part of the rotation including non-technical managers and analysts.
- I call this the Town Crier Method, and my key insight is that if everyone is getting woken up for issues, it's more likely for issues to get long-term resolutions. This even includes folks who can't directly resolve the issue but can help with contacting additional engineers and documenting the incident. No one wants to be responsible for a non-functional system.
- There is also additional camaraderie to be found when a wider group helps resolve issues. It visualizes to stakeholders and non-technical team members these critical incidents, in a clear and immediate manner.
- Thirdly, the larger an on-call rotation is, the lighter the impact is on everyone's free time yearly. The difference between a 10 and 20-person weekly rotation with a primary and backup is being on-call for 36 days a year, instead of 72.

Photo by Marvin Meyer on Unsplash

How to Replace Oracle JRE with Amazon Coretto JRE on Windows Desktops

Camille Clayton — Mon, 23 Dec 2019 18:46:22 +0000

2019 is coming to a close, and this quick guide will get your JRE cleared up for the next decade

Starting in 2018, Oracle began charging for Java SDK and JRE licenses for business use.

If you as a freelancer or your company, are using the Java Runtime Environment (JRE), but not the Java Development Kit(JDK), you may have been unaware of this change. Not to worry, multiple companies and projects are offering compatible versions of JRE and JDK.

The main player in this space is the OpenJDK project, but there are others including Amazon Coronetto, which we'll walk through using today.

To download the latest version of Amazon Coretto's JRE 8 head to this link.

From that page download the zip file that corresponds with your version of Windows, either x64 or x86. Most people need x64 but if you don't know which to choose, here's how to find out.

Going forward the instructions will be for x64, but should be very similar if you chose x86.

Extract the zip file to C:\Program Files\Java, this will extract a folder named jre8.

Now go to your Start Menu, and find Java, and then choose Configure Java.

It will open a window, and you should then choose the third tab that says Java.

Then click the View button.

On the next window under User choose Find.

Navigate to C:\Program Files\Java\jre8\ then choose Next and it should find bin\javaw.exe, and then choose Finish.

Back on the View Window, unclick the older Oracle JRE versions.

Choose Ok.

You can then test using a Java applet or program, to see if the JRE is working.

Hopefully this document helps you out, if you have any questions feel free to find me on Twitter @camofclay.