DEV Community: Lauri Hänninen

Analytics as Code: Managing Analytics Solutions Like Any Other Software

Lauri Hänninen — Tue, 08 Mar 2022 21:11:53 +0000

DevOps and CI/CD principles revolutionized software development. Anyone who wants to deliver high-quality software frequently and reliably uses these best practices. We use version control, write automated tests, and automatically deliver code from initial development to production to meet the demands of today’s world, where agility and speed have become critical competitive advantages.

These best practices are possible because we control the source code. If we couldn’t work with the code, we couldn’t make incremental changes to respond to the growing expectations and demands of our end users. But when we think of our analytics platforms, we’re not really able to access and manage the code we create when we build analytics with them.

So the obvious question is, if managing the underlying code and taking advantage of software development best practices have taken software development to a new level, why not apply the same techniques to analytics as well?

What is Analytics as Code?

Analytics as Code is the management of analytics using human and machine-readable configuration files. This means that our analytics solutions— connectors, semantic layer, dashboards, metrics, visualizations, user management, and other analytical objects — are transformed into a manageable piece of code. And this code should be treated exactly the same way as any other application source code.

The configuration files that define our analytics must be integrated into our version control systems to track, review, and monitor changes. With CI/CD platforms and testing tools, we can automate the integration and testing stages and deploy analytics to end-users faster, with higher quality and lower error rates.

High-level continuous integration process scheme. Adapted from https://medium.com/gooddata-developers/dataops-headless-bi-the-perfect-fit-2654a923ac01

This allows us to quickly innovate and experiment with new insights and make them available to end-users at an ever-increasing rate. We can minimize the time and effort required to turn requirements into solutions and improve and reuse them throughout the organization.

From a manual process to an easy-to-manage, reusable piece of code

Traditionally, all parts of our analytics are defined using the graphical user interface of the analytics platform. But because of a lack of openness and flexibility, the underlying code that we manually generate by clicking and dragging and dropping cannot be exported and managed outside the solution. This “what happens on the platform stays on the platform” approach has begun to limit analytics creation, management, and deployment. And it’s no longer a scalable solution as we strive to respond to today’s fast-paced world of analytics.

The traditional "what happens on the platform stays on the platform" approach.

Analytics as Code is based on modern analytics tools that support the import and export of all underlying metadata — in a declarative format — and provide open APIs to automate the ongoing delivery process. When we can export human-readable configuration files from our entire analytics solution, we can use both the platform interface and our favorite IDEs to manage the code and leverage best software practices. As a result, analytics becomes an easy-to-manage, reusable piece of code.

When analytical tools provide open APIs and support the import and export of all underlying metadata.

Examples of Analytics as Code configuration files

Below is a metric configuration file imported from an example analytics platform — GoodData.CN Community Edition — via open APIs. As we can see, the Total Sales configuration is both human and machine-readable:

Metric config file

In this exported configuration file for the Total Sales by Year visualization, we use the created Total sales metric (a reference to the metric in line 17) and slice it by year (line 37). The visualization type — column chart — is specified in line 51.

Visualization config file

Once we have a visualization, we can create a dashboard around it. In the following dashboard configuration file, we specify the layout and visualization (a reference to the created visualization in line 39).

Dashboard config file

Below we see what the created dashboard looks like on the platform. If we make any changes to the configuration files above — e.g., update the metric, change the type of visualization, or add a new visualization to the dashboard — we can import the files back to the platform, and the solution will update accordingly.

Created dashboard - GoodData.CN Community Edition

If you are interested, the complete config file of this simple example— data connector, physical data model, logical data model, users and user groups, and all previously displayed objects —can be found here: Configuration file.

Advantage of Analytics as Code

Analytics as Code makes managing and deploying analytics more efficient by dividing analytics into reusable code snippets and utilizing the same principles we use to scale up our other software. Here are some of the benefits that Analytics as Code offers:

Versioning

When we use configuration files, we can version the entire analytics solution and each object in it. Thus, all parts of our analytics are subject to source control, just like any other code.

CI/CD and Collaboration

Our data engineers and analysts can work simultaneously with different parts of the solution — semantic layer, metrics, dashboards, or anything else — and write automated tests to ensure that the logic we use works as it should. They don’t have to worry about breaking the work of others when they push updated versions into production.

Reusability

We can divide our analytics into modular code components, so our analytical objects become reusable code snippets that can be shared among teams. There is no need to re-create visualizations or metrics for different use cases, as we can reuse existing configuration files.

Consistency

Because the configuration files serve as a single source of truth, Analytics as Code ensures consistency across the organization. It ensures that everything works the way we want it every time we deploy or update our analytics.

Speed and Quality

We can make incremental changes to the code and quickly deploy updated analytics versions. The faster we develop and deploy our analytics, the higher the quality because we can deploy smaller snippets of code that are much easier to test. And to complete the process, we can quickly gather feedback on changes and respond to them immediately.

Automation

Declarative configuration files, along with open APIs, allow us to automate hideous manual tasks like (de)provisioning of new tenants and dashboard, metrics, and visualization creation. They also make it possible to programmatically change the configuration of our analytics solution.

Summary

The concept of Analytics as Code is simple; we should treat our analytics in the same way as any other software. This approach complements the functionalities offered by our analytics platforms and helps us move out of the current situation where we are at the mercy of platforms in terms of how we build and manage our analytics.

It’s time to turn our analytics into an easy-to-manage, reusable piece of code while leveraging software development best practices. By doing so, we can scale our analytics like modern applications, and ensure that we deliver data into people’s hands faster, more reliably, and more agilely so they can use it better for what it is intended for — to make better decisions.

About the Author: I'm Lauri Hänninen, Product Marketing Lead at Trezor. I specialize in translating complex technology, from crypto hardware security to B2B SaaS, into stories people actually understand.

You can find my full professional portfolio at laurihanninen.com.

Headless BI: Metric Standardization in Action

Lauri Hänninen — Tue, 08 Mar 2022 19:36:53 +0000

Metric standardization is a hot topic at the moment. Companies are deploying various solutions — metrics stores, metrics layers, and headless BI platforms — to provide consistent metrics to all of their data tools to avoid the danger zone of inconsistency.

This article provides a simple example of metric standardization, where different data consumers — SQL client, data science IDE, BI platform, and React application— access a headless BI platform, consume the same metrics, and achieve consistent results.

What is headless BI?

Headless BI means we separate the analytical backend and computing from consumption. This decoupling allows us to expose the universal semantic layer to multiple data tools via APIs and standard protocols.

Because all data consumers thus have access to a single source of metrics, our data engineers, analysts, and end-users can work with consistent metrics — with the same meaning for everyone — with the tools of their choice.

Setting up the headless BI platform

This article uses GoodData.CN Community Edition to introduce the concept of headless BI. GoodData.CN CE runs on our local machines as a container, and we will configure it with the GoodData Foreign Data Wrapper (FDW) needed for the headless BI use case.

GoodData.CN CE

To follow this article, you can download GoodData Python SDK, which contains a docker-compose file, and run the following command in the root folder:

$ docker-compose up -d

The docker-compose command starts both the GoodData.CN Community Edition and GoodData FDW containers and loads predefined analytical objects — data connector, semantic model, metrics, visualizations, and dashboard— into GoodData.CN.

Once the containers are running, let’s go to http://localhost:3000/ and log in to the platform.

User: demo@example.com
Password: demo123

Below is the logical data model for the Demo workspace created with the docker-compose. Later, this model and the Revenue metric are exposed to external data tools.

Logical Data Model — GoodData.CN (image by author

The predefined analytical objects also contain a Revenue metric. The metric uses another metric —Order Amount that calculates the income of all orders—and counts revenue only from delivered orders (order status is not Returned nor Canceled).

Revenue metric—GoodData.CN (image by author)

Below is the Order Amount metric used in the Revenue metric:

Order Amount metric — GoodData.CN (image by author)

On the Analyze tab, we can create a simple table that slices the revenue with regions. The results will serve as a benchmark, as we will re-create them with different data tools in future chapters.

Revenue by Region — GoodData.CN (image by author)

GoodData Foreign Data Wrapper

GoodData Foreign Data Wrapper is a PostgreSQL foreign data wrapper extension. It is built on top of multicore, and it makes GoodData.CN’s metrics, calculations, and data available in PostgreSQL as tables.

We can connect to the running PostgreSQL:

From console using psql --host localhost --port 2543 --user gooddata gooddata123
From any other client using JDBC string: jdbc:postgresql://localhost:2543/gooddata

    Username: gooddata
    Password: gooddata123

Once connected, we can work with GoodData.CN Foreign Data Wrapper. At first, we need to define our GoodData.CN server in PostgreSQL.

GoodData.CN server in PostgreSQL

Next, we will import the entire semantic model into a special compute pseudo-table. Doing SELECTs from this table will trigger the computation of analytics on GoodData.CN server based on the columns that we have specified on the SELECT.

The compute is called pseudo-table for a reason. It does not adhere to the relational model. The columns that you SELECT map to facts, metrics and labels in your semantic model. Computing results for the select will automati
cally aggregate results on the columns that are mapped to labels in your semantic model. In other words cardinality of the compute table changes based on the columns that you SELECT.
― GoodData Foreign Data Wrapper Documentation

Import semantic model into the pseudo-table

Consuming the standardized revenue metric

Currently, the revenue metric is used only on the GoodData.CN platform. Let’s see how to access the semantic model and consume the metric with other data tools.

SQL clients

First, the SQL client—DBeaver in this case— needs to be connected to GoodData FDW.

Connection type: PostgreSQL
Host: localhost
Port: 2543
Database: gooddata
Username: gooddata
password: gooddata123

Database connection — DBeaver (image by author)

Once the connection is ready, we can write an SQL query to calculate the same Revenue by Region results created earlier in GoodData.CN.

select customers_region, revenue from demo.compute;

Revenue by Region — DBeaver (image by author)

Data science IDEs

To work with Jupyter, let’s start the notebook server from the command line:

$ jupyter notebook

Next, we will use ipython-sql to connect to the FDW, run the same SQL query used with DBeaver, and print the Revenue by Region results.

Revenue by Region — Jupyter (image by author)

BI platforms

As a BI platform, this article uses Metabase, and we will run it locally as a container. The following command starts the Metabase container (note that the original port is changed from 3000 to 12345 because GoodData.CN CE uses the port 3000):

$ docker run -d -p 12345:3000 --name metabase metabase/metabase

Once the Metabase container is running, we need to create a network to connect the container with the FDW container because—as you guessed—everything is still running locally.

$ docker network create network
$ docker network connect network metabase
$ docker network connect network gooddata-fdw-container-name

Next, we can connect Metabase to the FDW pseudo table with the following details and credentials:

Host: host.docker.internal
Port: 2543
Database name: gooddata
Username: gooddata
password: gooddata123

Database connection—Metadata (image by author)

When the connection is complete, we can again use the same SQL query to compute the Revenue by Region results.

Revenue by Region — Metadata (image by author)

React applications

For this part, I created a React application using GoodData.UI accelerator toolkit. It is a CLI-based tool that guides you through creating the application step by step in your terminal application. The tool creates an application that is ready for use with none or minimal additional configuration needed from our side.

To start with the React project, we run the below command in the terminal and follow the instructions provided by the CLI.

npx --ignore-existing [@gooddata/create-gooddata-react-app](http://twitter.com/gooddata/create-gooddata-react-app) --backend tiger my-app

Once the build is finished, we need to go to the generated directory and start the app with the yarn start command.

cd my-app
yarn start

Then, we follow the directions on the main page and make the following edits to the src/constants.js file.

Constant.js config — React app

Next, we will generate human-readable JavaScript identifiers for the data model objects, which will later be used in the code. First, we export the GoodData.CN authentication token environment variable (the token is the same for all GoodData.CN CE installations), and then run the following command:

export TIGER_API_TOKEN=YWRtaW46Ym9vdHN0cmFwOmFkbWluMTIz
yarn refresh-ldm

Once the human-readable names for the data model objects and other metadata are generated, we can edit the src/routes/Home.js file to match the following code to create a Revenue by Region table:

home.js code for Revenue by Region table—React app

When we return to the browser and go to the Home tab, we see the embedded Revenue by Region results.

home.js Revenue by Region — React app (image by author)

Comparing the results

For ease of comparison, I combined all the results of the previous steps into the image below. As we can see, all tools accessed the same semantic model, consumed the same Revenue metric, and calculated exactly the same results.

Standardized Revenue metric across various data tools (image by author)

Summary

The purpose of this article was to show how easy it is to start standardizing metrics. Standardization means that all our metrics are defined in one place and can be consumed by different data tools, such as SQL clients, Data science IDEs, BI platforms, and applications.

With headless BI, the standardization is achieved by decoupling the analytical backend and computing from consumption and exposing the semantic layer via APIs and standard protocols. Thus, we can work with consistent metrics —with a shared understanding of what our data means— using the tools familiar to us.

You can find my full professional portfolio at laurihanninen.com.

DEV Community: Lauri Hänninen

Analytics as Code: Managing Analytics Solutions Like Any Other Software

What is Analytics as Code?

From a manual process to an easy-to-manage, reusable piece of code

Examples of Analytics as Code configuration files

Advantage of Analytics as Code

Versioning

CI/CD and Collaboration

Reusability

Consistency

Speed and Quality

Automation

Summary

Headless BI: Metric Standardization in Action

Table of Contents:

What is headless BI?

Setting up the headless BI platform

GoodData.CN CE

GoodData Foreign Data Wrapper

Consuming the standardized revenue metric

SQL clients

Data science IDEs

BI platforms

React applications

Comparing the results

Summary