Professional literature as a way to improve your analytics skills

Iaroslav Svet — Thu, 07 Sep 2023 17:53:55 +0000

This article is intended to reset the priorities for analysts from hard skills and frameworks to a broader understanding of the value of analytics. It will be helpful to data professionals of any level. As in the good old days, reading books is the best way to understand things better.

Online education has revolutionized the way we learn now. The process is as democratized as it never was; millions of users have access to a variety of courses. It is a significant trend that is difficult to overestimate. There are several things that could be improved with this approach, though. The first one is that the increased number of programs makes it more challenging to choose a good and trustworthy one. The second significant problem - the courses are usually short, basic, and very practical. This format converts better than others. The students who did such studies typically tend to think about the profession as a set of actions rather than trying to get deeper.

Analysts, like many other IT specialists, are also very often those who learn by doing. Learning by doing is an excellent way to boost your knowledge. But this boost may be much faster and have much higher quality if supplemented by previous generations' wisdom.

It is worth reading to enhance the knowledge from online courses or structure your knowledge in analytics gained in your workplace.

To be honest, there are not so many previous generations of data analysts in the current understanding of the term. The term data-driven has only been used in the business context since the early 2000s. It means that there are only 20 years of literature on the topic. But several books are a must-read for any analyst and head of analytics especially.

If you don't have time at all, take two books from practitioners - two books by DJ Patil, the second one co-authored with Hilary Mason: "Building Data Science Teams" and "Data-Driven - Creating a Data Culture" - both published by O'Reilly. Thus, you will have a chance to understand, in general, what a good analytics team should look like, what it should do, and how the decisions should be dated in a company.

If you have more time - please refer to the most cited works about data-driven decision-making - such pillars as Davenport's "Competing on Analytics" or papers by Brynjolfsson. These fundamental works mean the same for modern data analytics as Odyssey for contemporary literature.

If these two are Odyssey, Anderson's "Creating a Data-Driven Organization" is Uliss. Anderson builds over all the previous literature and perfectly explains all dos and don'ts.

Having read these books, you will have a clear vision of where you want to move your organization from the point of view of data usage for decision-making and will avoid some of the pitfalls that can wait for you in this way.

Rethinking your AB testing

Iaroslav Svet — Mon, 08 Aug 2022 12:56:33 +0000

Introduction

In the last years AB-testing has become a default way to take data-driven decisions about the product development in many companies. There are teams of analysts working together with product managers to take all the crucial decisions.

If you are an analyst or lead an analytics team you know that the experiments are not always perfect. As any regular process the process of AB testing has some common pitfalls and areas for optimization. In this article I am telling about my experience in a big tech-company with good established processes, the problems we faced and the ways we have solved them. It focuses rather on organizational part of the process than on technical or mathematical side of AB testing. The latter will be covered in a separate article.

Process description

The process described in this paper is AB Testing of new features in user products. The Company operates in the technology sector and runs dozens of popular internet sites and mobile apps. All the products are constantly being improved. To make sure only changes that increase the product key metrics are launched, the product team runs AB tests, comparing the version before changes and the updated version of the website or app. The current procedure assumes that the product manager of the new feature is an internal customer of this process. The process can be simplified and described as presented in the picture:

A manager launches a new feature and asks an analyst to perform an AB test. If the AB test data is valid and the key metrics in the test group are higher than in the control group, the feature is launched into production. In another case, there is an iterative process of problem detection. The latter may take much time.

The general concern of all stakeholders is that it takes too long to assess the results of an experiment.

To get a better understanding of product managers’ view on the issue we can perform an I/P analysis to understand which areas have problems from the point of view of internal customers, what are our strengths and weaknesses.

X axis is the importance of the process feature to the customer
Y axis is customer’s assessment of how good is it done now

After visualising the I/P matrix, it becomes evident that the priorities are flawed due to the wrong incentives of the product managers (who are supposed to be internal customers). Product managers are interested in launching fast all the features they were working on (caring about sunk costs). They care less about the precision of the numbers and the quality of the product. Analysts are treated as an external resource; optimisation of their labour is not a priority.

It would be helpful to redraw the matrix from the point of view of the business owner.

From the point of view of the business owner, the most crucial goal is to keep a high quality of the product while launching new features fast. The time needed to test features matters much less than the time to launch the feature.

A bigger process

The process of AB-testing itself is not separate. It is a part of the process of launching a new feature.

Redefinition of the process customer and the borders of the process itself let us think broader about the potential ways to solve the problems we face. The process that we actually want to optimize is launching new features.

We can ask 5 Whys to understand better the problems of this bigger process:

In our case, defects are the most critical problem of the AB testing process. The quality of the product delivered to an AB test might sometimes be not good enough, which leads to several iterations of problem detection and explaining.

It can be fixed by improving the QA process, which is interesting because the potential optimisation for AB-testing does not depend on the analytical team at all.

DEV Community: Iaroslav Svet

Professional literature as a way to improve your analytics skills

Rethinking your AB testing

Introduction

Process description

A bigger process