DEV Community: Sam Blausten

Update the Backstage catalog instantly without touching any YAML

Sam Blausten — Tue, 22 Oct 2024 08:59:18 +0000

Updating an entity in the Backstage catalog generally means manually updating a YAML file in a repository somewhere and getting it merged to the main branch.

For many, this manual process can feel like a series of hurdles—switching contexts, waiting for pull request reviews, and depending on other team members or systems.

This friction increases significantly when you scale across hundreds or thousands of repositories. For example, changing a group name might require chasing after numerous teams to raise and merge PRs, turning what should be a simple update into a months-long process.

Having to edit YAML makes keeping your catalog up-to-date a challenge, let alone enriching it with additional data or plugins like PagerDuty.

How long does it take to add a PagerDuty plugin?

Lets use the PagerDuty plugin as an example. Suppose you want to add PagerDuty monitoring to a Backstage entity representing a backend service. All you need to do is add an annotation to the YAML file with the relevant service ID.

The quickest way to edit the YAML file might seem simple: open GitHub, click the pencil icon on the About card, and start editing.

Manual Process in GitHub

Once inside GitHub’s web editor, you can make the change, commit it to a new branch, and open a pull request.

Sounds straightforward, right? But here’s where it gets tricky. To add the PagerDuty plugin, you need the correct service ID, which requires logging into PagerDuty, searching for the service that matches your entity, and then copying the ID from the browser’s address bar (since it’s not easily accessible in the UI).

In GitHub’s case, we can edit this immediately, commit it to a new branch and open a pull request rather quickly all in the web editor.

But to find the correct Service ID we must first go to PagerDuty. Hopefully we can log in and access the available services, and then search for the correct one using the names in the Component YAML.

Now, after locating the ID (in the address bar) and adding it to the YAML file, you still need to commit the changes, submit a PR, and wait for it to be reviewed and merged. Depending on the team’s review cadence, this could take days or longer.

Multiply this by hundreds of teams and thousands of repositories, and the process becomes incredibly slow and frustrating. Minor updates can take weeks or months to implement at scale.

A Better Way: Entity Updates with Roadie

At Roadie we’ve built UI based tools that allow you to set things like PagerDuty annotations with a few clicks. Instead of seeing a missing annotation card you’ll be able to select from a list of PagerDuty services and update the entity immediately, all from the same page in the UI.

Bulk updates via UI

But what if you need to update multiple entities at once? Roadie also provides a bulk editor that allows you to make updates across many entities using a simple table format. This dramatically reduces the manual effort required to keep your catalog accurate and up-to-date.

Behind the Scenes: How It Works

Our UI-based editors are powered by a backend feature we call Fragments. Fragments are partial entity data stored in a database table and merged into the existing entities in the catalog by a processor. These fragments can be updated through our UI or via API.

Leverage external APIs to make things easier

The custom UI we’ve seen earlier in this post also leverages external API’s to provide a dropdown of available options.

In this example we’ve used PagerDuty’s API to fetch a list of services using the API token already added to Backstage to make your PagerDuty plugin work. This call is made via the Backstage proxy which is set up to point to PagerDuty using your API Token on the backend.

Script bulk changes via API

However, sometimes bulk editing of metadata in the catalog is best scripted, and in this scenario, Roadie users make use of our Fragments API to make changes to entities, such as changing the ownership of a group’s entities when the group gets absorbed into another group, or updating any relationship references to a user entity when that individual leaves and someone else takes their role.

We’re not the only ones who’ve done something like this - companies like Twilio and others have built similar fragment functionality so that they can script updates to there catalog via API and free up platform teams to help keep the catalog data rich and accurate.

Reach out to us on Discord or our website’s chat messenger if you want to hear more about this and other improvements Roadie has made to the Backstage experience.

Easier Relationship Mapping in the Backstage Catalog

Sam Blausten — Mon, 07 Oct 2024 07:45:17 +0000

In the Backstage software catalog, relationships provide a glue between the different software, systems, resources and people in your organisation. They can allow you to easily answer questions which otherwise might be much harder to find out in a large or rapidly growing organisation.

Plugins like the Catalog Graph Plugin allow powerful visualisation of these relationship graphs.

Similarly, relationships can become a core piece of information displayed on entity Overview pages.

Relationships can be added in one of two ways - via a YAML file update in an SCM repository, or in a third party source like GitHub Teams or Okta. A YAML update looks something like this:

kind: Component
metadata:
    name: identity-backend
spec:
    ...
+   dependsOn:
+       - component:default/users-backend
+       - resource:aws/identity-backend-ec2

Restrictive Relationships

There are a very limited and restrictive set of allowed relationships between different Kinds in Backstage by default. Knowing what is allowed requires looking up the Backstage documentation each time to check the schemas.

For instance, a Domain cannot “depend on” anything in the default Backstage relationship model. However, maybe you have a domain like Shipping that depends on another like Identity. The Identity domain exposes APIs for customer addresses that the Shipping domain breaks without access to. You might even want to directly model the hard dependency on specific external API’s inside the Identity domain so that its easy to identify issues and notify the right people in charge of the Shipping domain when something breaks in those APIs.

Confusingly, many of the default relationships terms that can be used in the YAML spec definition don not map directly to the actual relationship in the catalog. For instance if you want to add a partOf relationship for a Domain to another Domain, you would need to use subdomainOf . Adding it as an explicit partOf field would not work unless you add that in the extended processor as we’ll see shortly. Each Kind has its own differing semantics for each relationship type, which needs to be referenced and checked to ensure a relationship is correctly added.

Expanding available relationships

Luckily its an easy engineering task to add a processor that can handle more expansive relationships and even new terms for those relationships such as manages/managedBy for Users. This processor would run in addition to the BuiltinKindsEntityProcessor that does the default relationship processing for the catalog and would look something like this:

import { CatalogProcessor, CatalogProcessorEmit, processingResult } from '@backstage/plugin-catalog-node';
import { Entity, getCompoundEntityRef, parseEntityRef, RELATION_DEPENDENCY_OF, RELATION_DEPENDS_ON } from '@backstage/catalog-model';
import { LocationSpec } from '@backstage/plugin-catalog-common';
import { RELATIONSHIP_MANAGED_BY, RELATIONSHIP_MANAGES } from '../constants';

export class ExtendedRelationshipsProcessor implements CatalogProcessor {
  getProcessorName(): string {
    return 'ExtendedRelationshipsProcessor';
  }

  postProcessEntity(
    entity: Entity,
    _location: LocationSpec,
    emit: CatalogProcessorEmit,
  ): Promise<Entity> {
    const selfRef = getCompoundEntityRef(entity);
    function doEmit(
      targets: string | string[] | undefined | null,
      context: { defaultKind?: string; defaultNamespace: string },
      outgoingRelation: string,
      incomingRelation: string,
    ): void {
      if (!targets) {
        return;
      }
      for (const target of [targets].flat()) {
        const targetRef = parseEntityRef(target, context);
        emit(
          processingResult.relation({
            source: selfRef,
            type: outgoingRelation,
            target: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
          }),
        );
        emit(
          processingResult.relation({
            source: {
              kind: targetRef.kind,
              namespace: targetRef.namespace,
              name: targetRef.name,
            },
            type: incomingRelation,
            target: selfRef,
          }),
        );
      }
    }

    if (entity.kind === 'Domain') {
      doEmit(
        entity.spec?.dependsOn as string[],
        { defaultKind: 'System', defaultNamespace: selfRef.namespace },
        RELATION_DEPENDS_ON,
        RELATION_DEPENDENCY_OF,
      );
      doEmit(
        entity.spec?.dependencyOf as string[],
        { defaultNamespace: selfRef.namespace },
        RELATION_DEPENDENCY_OF,
        RELATION_DEPENDS_ON,
      );
    }

    if (entity.kind === 'User') {
      doEmit(
        entity.spec?.managedBy as string,
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGED_BY,
        RELATIONSHIP_MANAGES,
      );
      doEmit(
        entity.spec?.manages as string[],
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGES,
        RELATIONSHIP_MANAGED_BY,
      );
    }

    if (entity.kind === 'Group') {
      doEmit(
        entity.spec?.managedBy as string,
        { defaultKind: 'User', defaultNamespace: selfRef.namespace },
        RELATIONSHIP_MANAGED_BY,
        RELATIONSHIP_MANAGES,
      );
    }

    return Promise.resolve(entity);
  }
}

Questions to consider

There is a rational that restricting the available relationships for each kind in the catalog prevents misuse of relationships by users adding things to the catalog. However it is worth bearing in mind that relationships in the catalog are mostly just syntactic sugar over the same link between two entities. There is nothing different about dependsOn and partOf in Backstage except the perceived meaning of these terms.

Educating users

Easily accessible documentation on standards and definitions is the best way to educate contributing users to your Backstage catalog. Backstage has its own schema docs and descriptions on existing relationships but you can also host internal documentation in a more compact format that is easier to reference quickly.

What we did to fix it

At Roadie, we’ve added expanded relationships for all kinds in the catalog to make it easier for people to correctly add relationships using the terms they understand. We’ve documented them in a compact format and are working on a UI based editor for these relationships so that users do not have to manually edit each YAML file and go through a PR process to build up a comprehensive mapping of dependencies in an organization.

We've also added a new card that can show all relationships for an entity, regardless of what kind of relationship it is in list format.

Contact us via Discord on on this website's chat to find out more about what Roadie can help with or to talk to our engineers about this topic or other issues you might be having adopting Backstage.