DEV Community: Thomas Gerber

Git + Jira Analytics in 10 minutes on your laptop

Thomas Gerber — Fri, 18 Nov 2022 05:41:06 +0000

Product and Engineering Managers, rejoice! You can now get Git and Jira analytics on your laptop in 10 minutes! It is free, open-source, and all the data stays with you!

Explore and Leverage all your engineering operations data (epics, issues, pull requests, builds, releases) to make informed decisions.

Docker Desktop and git required. Open your favorite terminal and launch:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/faros-ai/faros-community-edition/main/install.sh)" _ --source devto_analytics

Open source code
Documentation
Faros Community Slack

Say goodbye to stale spreadsheets and rigid metrics!

Making the case for the EngOps data fabric

Thomas Gerber — Wed, 18 May 2022 16:40:42 +0000

As an engineering leader, I've worked with data all my life. In fact, most recently, I was in charge of the data layer of Salesforce Einstein, Salesforce’s AI platform. Even with all the data expertise in our organization, it was strikingly obvious that the engineering function in most companies my peers and I worked for, have not been able to fully leverage all the data in a unified manner. The problem - data is often scattered across disparate systems. A better data-driven approach is a must if we want to move from gut-feeling and guesswork to intelligent actions that impact real business outcomes.

All other functions have great data fabrics:

Sales teams have Salesforce. They have sales pipelines, automated data enrichment processes, revenue predictions, and SalesOps, which is now a very well understood role.
Marketing gurus have Segment & Google Analytics. They can track visits, attribute them to campaigns, and can calculate cost of leads to the last dollar.
Product Managers have Amplitude. They can map customer journeys, predict churn and LTVs, and segment audiences into personas.

On the other hand, engineering usually does not have anything similar. That is because compared to other functions, software engineering is an artful craft, one that is rapidly evolving. As such, choices of tools are made locally, in a bottoms-up fashion, which leads to massive fragmentation of data. How many CI/CD systems does your engineering organization use? How many CRMs does your Sales organization use?

In many cases, engineering leaders are often forced to cobble data together in spreadsheets in order to perform meaningful analysis. Take Lead Time for Change as an example, one of the 4 DORA metrics that research suggests is meaningful to track for engineering organizations: not only do you need to ETL data from multiple systems (commits, pull requests, build, artifacts, deployments) to compute it, the collected data needs to link properly together. You need a robust data system to gracefully deal with missing data and out-of-order data ingestion. Most likely, you will also need to capture changesets for your deployments. A very tall order. As the old saying goes, the shoemaker's child always goes barefoot.

Even though metrics vendors may alleviate that pain somewhat, it is not sufficient. The metrics those tools capture and surface are fairly static, and their domain of applicability is limited. Notice that the products mentioned above have analytics as a foundational capability: you can measure and track anything you want on your data. What you don’t know can hurt your teams - and your bottom line.

I want to make the case that engineering organizations similarly need a new data fabric centered around EngOps; a fabric that should of course cover the main software engineering value stream elements (Tasks, Pull Requests, Incidents, Builds, Deployments, and more), but can also extend and simplify compliance, recruiting, employee satisfaction, and OKRs.

Data fabrics usually have, at a minimum, the following characteristics:

Practical and Connected: Value comes from how well the data is modeled after the world - Lead / Opportunity / Account in Salesforce; Campaigns / Sources / Mediums in Google Analytics; User Sessions in Amplitude. Great data models have relationships properly connecting events and entities together for increased value: for example in Amplitude, a user can be in the ‘new’, ‘current’, ‘dormant’ or ‘resurrected’ based state on their behaviors. For EngOps, that modeling and how the different data elements connect is especially critical given how many different systems are at play.
Actionable and Extensible: Data can be analyzed, aggregated, and visualized any way the user sees fit. It can be used for automation purposes through APIs and exported for further processing. It can be extended by the user: for example custom objects / fields in Salesforce; properties in Segment / Amplitude.
Trusted and Intelligent: Data can be observed at the most granular level: for example, Segment, Amplitude and Google Analytics have live debuggers/feeds to introspect data as it changes or arrives in the fabric. Data is also automatically improved, through inferences on how it connects and imputations of values; those improvements are documented and remediable.

Now, here are a few concrete examples of what an engineering leader could do simply (minutes or hours, not days) with such an EngOps data fabric:

Dive into the data to craft meaningful policies and investment objectives that impact the business - and then track corresponding Key Results:
- Is onboarding new engineers going better over time, or worse? Is remoteness making onboarding less effective?
- Is the lead time per integration decreasing? Where is the bottleneck? Does each integration require changing the underlying APIs or are those durable?
- How do meetings and interviews impact code delivery?
Automate based on a trusted, transparent metric:
- Automated deployments if the Change Failure Rate of the application is low enough
- Automatically adjust the type of under-utilized cloud instances
- Collect compliance evidence and enforce policies automatically

Clearly, you shouldn’t be focusing on building such an EngOps data fabric. It is challenging to build and not your core business. This is what we build at Faros AI: the connected engineering operations platform. We want to unlock the power of all that EngOps data for your organization. You can check out our open-source version: https://github.com/faros-ai/faros-community-edition.

(originally posted here)

Why computing DORA metrics is hard

Thomas Gerber — Mon, 18 Apr 2022 16:37:52 +0000

The DORA metrics are a set of metrics that measure the quality and velocity of software delivery of an engineering organization. By measuring and continuously iterating on these metrics, engineering teams can deliver better software to their customers faster, and achieve significantly better business outcomes.

So why aren’t those metrics widely available in your engineering organization? Because they are difficult to compute. Here are a few of the hurdles we faced when we were building Salesforce Einstein, and the reason we started faros.ai. Keep these (and us!) in mind when you start this journey.

Let’s start with the time to restore service metric. Good news: you can probably get this metric from your incident management system. Assuming you are OK with not having a single pane of glass for your metrics. But don’t get ahead of yourself!

Now let’s look at change failure rate. This is a metric that combines your incident and your deployment data. This means you need to extract and load data from both sources somewhere to compute the metric. You cannot just get it from one of your systems. That’s just the first difficulty. Next you need to correlate your incidents with your deployments.

Then, let’s take a look at weekly deployments. It is common for Engineering organizations to have several CI/CD systems. The data from the different systems will need to be modeled and normalized. More fun!

Lead time for changes ups the ante: not only do you need data from multiple systems (commits, pull requests, build, artifacts, deployments), that data needs to link properly. Your system will need to be robust enough to gracefully deal with missing data and out-of-order ingestion. Most likely you will also need to impute changesets for your deployments.

Finally, if you overcome those difficulties, you will want to roll up those metrics along organization lines, slice and dice by application, by stage, and over time. All of a sudden, you have to maintain a service catalog, deal with hierarchical roll ups and drill downs, …

TLDR: DORA metrics are difficult to compute; don’t build them from scratch. Use tools and products that deal with these difficulties to get to value faster. Check out Faros Community Edition.

How we Learned to Stop Worrying and Love Data Movement with Airbyte

Thomas Gerber — Mon, 11 Apr 2022 17:17:27 +0000

Faros Community Edition (CE) is an open-source engineering operations platform that connects the dots between all your operational data sources for a single-pane view across the software development life cycle. Airbyte powers Faros CE.

In this post, we will explain:

Why Faros CE needs ELT
Why we chose Airbyte
How we use Airbyte and contribute back
How we let our users unleash even more of Airbyte’s power

1. Why Faros CE needs ELT

Faros CE provides API, BI, and automation layers for engineering operations data (from Pull Requests to Incidents). With Faros CE, you can answer questions about bottlenecks in your CI/CD pipelines, issues in your onboarding processes, progress towards your goals, with data that is fresh, connected, queryable, and most importantly, all in one place.

But before any of this can be achieved, data needs to be ingested into the Faros CE platform from many different sources (GitHub, Jira, CircleCI, PagerDuty, …) in order to properly connect and analyze that data. Faros CE decided to leverage Airbyte for that ELT.

2. Why we chose Airbyte

We didn’t want to spend our energy working on data movement. Instead, we want to focus on our data schema and its useful applications for our community. We hence had to pick a data integration platform.

There are 3 reasons Airbyte stood out:

Airbyte has a vibrant community; just check those stats out (powered by Faros CE)!
Airbyte has become the de-facto standard for data movement
Airbyte already has a rich catalog of connectors that we can leverage (GitHub for example)

3. How Faros CE integrates Airbyte and how we contribute back

To integrate Airbyte, we first created the Faros destination connector. It knows how to send data from sources into the platform, and takes care of mapping the various entities being extracted against our rich, connected data schema. Because of the great Airbyte specification, that code is compact and focused on the value Faros CE provides (i.e. that mapping).

Second, we contributed to source connectors important to Faros CE (Jira) or new source connectors that didn’t exist yet (Jenkins, Customer.io, Harness, VictorOps, PagerDuty). Since we use Typescript, we decided to contribute our Typescript CDK. It is our way to strengthen the community while benefiting from it!

Finally, whether it is through docker-compose (local) or plural.sh (cloud), Airbyte is deployed as part of Faros CE, with pre-configured connections. Our users only need to enter credentials to their source systems, and can have data ingested in the platform in a matter of minutes!

4. How we let our users unleash even more of Airbyte’s power

Our users have direct access to the Airbyte instance that launched as part of their Faros CE platform. They have therefore access to its full feature set! In particular, they can create custom sources or contribute new sources to the Airbyte community with minimal effort!

Airbyte is the bedrock on top of which Faros CE is built. The product and its community enables us to focus on our core value. We are proud members of the Airbyte community.

BI for EngOps with Metabase

Thomas Gerber — Fri, 08 Apr 2022 17:04:46 +0000

What can you do with Faros CE thanks to Metabase?

Faros CE provides ELT, APIs, BI, and automation for engineering operations data. With Faros CE, you can answer questions about bottlenecks in your CI/CD pipelines, issues in your onboarding processes, progress towards your goals, with data that is fresh, connected, queryable, and most importantly, all in one place. Let’s dive in to show how Metabase makes that simple.

Let’s take lead time for example. Lead time is a software delivery metric popularized by the DevOps Research and Assessment (DORA) organization, that measures the time it takes for changes to go from idea to production. That number in itself is not sufficient, however. It is important to put it in context. And Metabase makes it simple to provide that context! In the Faros CE DORA Metabase dashboard, you can see

How your lead time compares to known industry benchmarks thanks to the gauge visualization with ranges
The evolution of lead time for each of your applications over time with a line chart
The breakdown of lead time by stage, so you can start to see where the bottlenecks are in your software delivery process thanks to stacked vertical bar charts.

Our users love to:

Zoom into the underlying data by simply clicking on the charts.
Create their own metrics and charts in Metabase from scratch or from our canned charts, without the need for SQL and with previews at every stage
Use X-Rays for fast insights in their datasets

This interactivity in Metabase creates a virtuous cycle of increased trust into the metrics and increased usage and exploration. And that cycle is easily bootstrapped.

Software Composition: building on the shoulders of giants

Thomas Gerber — Thu, 07 Apr 2022 15:51:25 +0000

When we decided to release Faros Community Edition (CE), the open source version of our SaaS product Faros AI (https://faros.ai), we decided to focus on 2 things: time to value, and making it as welcoming as possible to users and contributors.

As an open-source engineering operations platform, our data schema (which models everything from pull requests to deployments) is the keystone of our value proposition. We wanted to make it as useful to the community as possible. Which is why every capability of Faros CE leverages that data schema, AND is powered by best-of-class open source projects!

We rely on:

Airbyte for ELT
Hasura for our GraphQL API
Metabase for BI
n8n for Automation
dbt for transformations

As a result, the Faros CE codebase is compact: the data schema, some initialization scripts that compose those services together, and some pre-configured dashboards and endpoints.

Faros CE is up and running in minutes! And users can leverage the full power and extensibility of those best-of-class projects. For example one can:

Connect to new or custom data sources thanks to Airbyte’s ever growing source catalog and development kits
Enrich data ingested in Faros CE thanks to Hasura’s Actions feature
Get automatic insights into their Engineering Operations data thanks to Metabase’s X-Ray feature

Composition is truly the most powerful capability in software engineering. Yay functional programming?!

We welcome your thoughts on this and encourage you to try out Faros CE by heading over to our GitHub repo - https://github.com/faros-ai/faros-community-edition

Here is an example of what you can achieve in minutes!

Same visibility into your SDLC as the apps you develop!

Thomas Gerber — Tue, 05 Apr 2022 21:06:40 +0000

As software engineers, we understand how important it is to instrument our applications. Without Datadog, New Relic, Sumologic, we know that Engineering would grind to a crawl at the first sign of trouble. The same holds true for your Software Delivery Process, i.e. how you review, merge, build and deploy changes, resolve incidents and fix bugs.

With Faros Community Edition, you can now have unprecedented visibility into your Software Delivery Process.

Anyone used to tackling this problem quickly hits a wall: data is in a lot of different places and cannot always be easily leveraged. For example:

for Change Failure Rate, your incidents are in PagerDuty, your Deployments in CircleCI
GitHub only gives you metrics for a single repository, and nothing close to PR Cycle Time

At this point, you may think “all those systems have APIs! Let’s extract all that data and compute metrics ourselves! I know the perfect place to put it!”.

This is when the REAL fun begins!

Data Integration is a nightmare
Linking data is necessary for some metrics (like lead time for changes) but incredibly hard
If your teams use several systems (say CircleCI and GitHub Actions), you have to deal with data modeling and normalization
Self-serve can quickly become daunting, and maintenance is hampered

This is why we built Faros Community Edition.

Features to consider:

🗺 Rich Data Schema
Connected canonical models for the whole SDLC; 50+ entities, from tasks to deployments

🚰 Import from a variety of sources
Easy data import onto our models from Task Management, Version Control, Incident Management, and CI/CD systems

❄️ Flexible GraphQL API
Leverage imported data for automation / exploration in our canonical representation

📊 Preconfigured dashboards
View well known engineering metrics such as DORA and SPACE

🏗 Extensibility and shareability
Build and share custom metrics and dashboards

☁️ 💻 Container-based deployment
Run on your laptop, private or public cloud, with no external dependencies

Get started in 10 min, get your questions answered and finally ditch the spreadsheets!