DEV Community: Ivan

How to prioritise Data Science (and not only) tasks in 3D

Ivan — Tue, 14 Jul 2020 16:39:53 +0000

Data Science is a very vague term. Companies define it very differently and it depends on the field and size of the data team. Data Science can be a product, play a crucial role in product development, or be only in a supportive position. This leads to different types of Data Culture, communication between product and data teams and changes the way you prioritise things. I'm not going to focus on specific cases of prioritisation but give you a general framework I use both in my job and personal projects.

Prioritisation isn’t easy. There is no single axis which helps you to triage your backlog. But there are three which can be helpful. Here they are (I call it 3D):

Dependency. Is it required for other things?
Duration. How long will it benefit us?
Damage. What does it cost?

I'm not including "Benefit" or "Impact" here because it's obvious. If something isn't beneficial, why should you spend time and resources on it? Moreover, some things couldn't benefit you directly, or it wouldn't have an instant effect but something which will pay off in a year or more. Some things are required to other much bigger projects and identifying them is an important skill. It leads us to the first D.

Dependency is what helps you to build up a groundwork for the future. It has two subcategories - projects or tools and knowledge. Projects or tools are the bricks in your Data Science tech stack. If there's something that will help you to create something much bigger than you can do now, it should have higher priority.

I like the concept of The AI Hierarchy of Needs. You can use it to identify priorities inside the Dependency axis. All infrastructure is a dependency. You cannot do your magic if your database is not working or users don't send you data. You also cannot so your magic if your data is raw and full of anomalies or broken JSONs. And to be sure your data is always good you need Analytics - metrics, dashboards, monitoring, all of these. You should still consider other axes and don't get too deep into the infrastructure woods. Improvements have no limit. It's important to know when to slow down. What you need is just a reliable ETL flow and basic Analytics to track what's going on with your data. In short, you need a good data to do your magic. Everything which leads you to this comes first.

The second things are Knowledge and Learning. It isn't something that has an instant effect. It benefits in the long term. The more valuable skills you and your team has, the more opportunities to apply it you have. Sometimes knowledge is a requirement, sometimes an opportunity isn't clear. But it shouldn't be underestimated.

There's a set of skills which is necessary for every Data Scientist and any person who works with data. Investing in profound knowledge of fundamentals will pay off anyway. But spending weeks on a trendy framework could be a waste of time. Lack of knowledge makes you unable to explain important concepts to non-technical people which in reality is very important. You can do more mistakes in trivial things and just slow down the process.

The second D is Duration of the effect. You need to focus on things which won't be obsolete soon and will be able to benefit for months or years. Both projects and knowledge are suitable for this. Good examples are infrastructure, fast and efficient database, in-house developed frameworks, libraries and tools, knowledge base and many other evergreen things. Bad examples are patches, focusing a lot on curiosity-driven ad-hoc requests.

Treat everything you do as a data product. The key metric for the product is LTV which is depends on revenue (benefit) and retention (duration). So your goal is to maximize the compound impact of your work over a long period of time. This approach might not be suitable for everyone because can be too slow for very fast paces environments like startups. But even there building a strong foundation for the future is a very good thing. It's important to see the trade-off between what you need now and what you'll need in a few months.

The last D is Damage or Costs or Risks. Call it whatever you want but it means only one thing - the resources you need to spend on the project or task - either human resources, money or CPU time. Every of it has a money equivalent. Spending working hours or spending money always have an opportunity cost. Whenever you do something, you don't do anything else. That's why we prioritise stuff, don't we? If something is too costly, it may have low ROI just because of it and the project will not be profitable. Same for a data product, including internal.

You can come up with advanced AI model that will increase conversion by 20% but it will never be in production if you need a year to build it and a couple of engineers for supporting it further. That's why you need to start start with things you can easily validate. If you can build a prototype within a reasonable amount of time and then test it - half of the job is done. You already know whether it will pay off or not and it makes it very easy to convince your team, directors or C-level to select this for development and give all the resources you need.

The last rule, which wasn't in the list, is make more with less. Always do things which are easily scalable to get more benefit and almost don’t require to scale efforts and needed resources at the same time. AI is overrated, heuristics are underrated. If one if-else does 80% of AI's job then you don't need AI. Always put high-leverage things first. I see this as the main beauty of Data Science - making value from nothing. Understanding it makes prioritisation much easier (and sadder at the same time). When you realise that your job is not about doing fit/predict all day you'll look at it from a different angle.

Thanks for taking the time. If you like this essay, want to ask or suggest something, you can find me on Twitter. I appreciate feedback of any kind.

When Data Science turns into Homoeopathy

Ivan — Wed, 12 Feb 2020 15:44:47 +0000

Hey everyone! My name is Ivan. I'm a data scientist working in the game industry.

Recently I realised that there's too much stuff I can write about and I thought... Why not? Maybe you will learn something useful and benefit from it. The perfect case - we'll be able to discuss something together!

So, here is my first post originally posted in my blog Bias Distillery. Very excited to share it here.

WARNING: this post is not about real "homoeopathy". It's a metaphorical term I just use when people think that being a bit data-driven should fix their flawed workflow.

From the very beginning of my career, I've been working with data and people who also use it somehow. And this post is based on my and my colleagues experience working in companies which actively use data. I'll try to advocate of developing and sustaining strong Data Literacy and Data Culture in general. And why it's crucial for everyone who touches data. There are a few things I've realised too late and want to share with the world.

Where does the pain come from?

Let's start with a simple and very exaggerated example. Let's say it's a company which develops mobile apps.

Random person: We have a marketing campaign with 25 users from a new ad network could you please predict their LTV for us?

Data scientist: Seems that you don't have enough data so far, a variance must be too high. Buy more users and come back.

Random person: Variance, eh? We don't have time for this, we need to know results ASAP. Just give us your predictions, we'll decide what to do next.

Data scientist (thinks): it sounds like homoeopathy to me.

Data scientist: Alright, you know better.

So who is wrong in this situation? Now I would say - both. Unfortunately, my answer wasn't like that all the time.

At first, it's obvious "random person" wasn't data-savvy enough. In most of the cases, it's literally impossible to say anything about traffic source with only 25 users. And he should've known that sample size is very important you need to keep in mind working with data. And data scientist should have explained that and suggest a solution to this problem. Unfortunately, people sometimes cannot accept another solution and it creates problems for both sides.

Giving people data sometimes makes them feel like they already know what to do with it and have some kind of superpowers. But usually, they don't, at least not everyone.

Imagine your team have a model in production which make predictions on a daily basis. You read a ton of papers, articles and books, spend a few weeks or months to develop, validate and finally deployed it. You are sure that the model is 90% accurate. You deliver results via a simple dashboard or any other tool. What happens next?

The answer is "it depends", really. If it's a not completely automated decision-making system and somebody is using these predictions after they're ready, a very possible situation is a person to whom this data was intended may use it wrongly and the company will eventually lose money blaming data scientists that their model is wrong.

It happens because working with data is a skill. And each skill could be learned, no one has it initially. And I think, It is the responsibility of Data Scientist to make sure the clients are capable to work with data and teach if they don't... or keep them away from touching data.

Any data-based products, from ad-hoc analyses to advanced ML model predictions, shouldn't be limited to delivery of their results to a "client". The last step or, which is more likely, even the first one, to make sure that people who will work with the outcome of data analysis know how to do that properly. So, simple rule I follow now is "know your audience".

A bit about problems of democratized data

Some people say, that in a perfect world - anybody should be a Data Scientist. However, I would disagree with this statement because in this case, I would lose my job. Being serious, in a more realistic "perfect" world - anybody should be data literate at least. It's important for people of many professions, not only data people.

As I mentioned, working with data is a skill, and it's required to ask the right questions and look for correct answers. I'm not talking about wrangling huge datasets - no, not at all, even facing simple everyday problems like checking KPIs at the morning in the daily mail, generating product ideas or requesting some data analysis. In teams which have a data scientist, all other people are often data users as well. They make requests, wait for the analysis and make decisions. They also could use data from dashboards or any BI systems. It's a really great approach. However, sometimes it faces serious problems.

The lack of data-savviness and even basic knowledge of statistics or math creates problems for developing healthy Data Culture and makes it no sense to "democratise the data". By this, I mean making it accessible to anyone in the team or company. The culture code of many modern companies (especially tech or digital) is to make everyone make decisions based on data. It can be both in data-driven or data-informed way depending on how Data Culture is developed. It also questions transparency, communication and many other things which are also interesting topics but not a part of this post.

if data is democratised in a company with strong data culture and literacy it becomes a strong advantage for business and every team member. People are capable of generating the right hypotheses, asking the right questions and making the right decisions based on data. And as a consequence, business moves forward and everyone is happy.

If data is "democratised" in company with weak data literacy among team members it ends up with a mess and suffering for data scientists. They are bombarded with stupid questions, they are digging data which somebody asked out of curiosity and spending their time to explain people they work and all the time proving that data is not lying. It can be solved by the right task and project management in the data team, however, a pressure, requirements and expectations from the data team are very high in this case.

My colleagues and I were in both situations from time to time depending on a team or project we were working. There were situations when we were asked to explain what is median and why we use it or how to read scatter plot. I remember working with people who sent excel report back to me with a request to add one column which could be calculated from the other two on the same sheet and make a simple line chart from it. If these persons had basic skills of working with data, situations above would've not even existed. It would save a lot of time for both parties and improved the overall team's performance.

Data trustworthiness and "experts"

Things become more complicated when upper management also has no idea how to work with data and what they want to get from it. Expectations from data scientists are also very high in these companies. Managers ask DS to “find something interesting”, without pointing on a problem which they are willing to solve. Sometimes, they even hire data scientist only because it's a buzzword. Data scientists are not "magic unicorns" as people often think, they also need some time and help to understand the problem, think about the right questions, especially in an unknown domain. Hiring DS and telling him/her "SOLVE MY PROBLEMS" it's not a good strategy.

Tweet by @threddyrex

Many managers are often sceptical about data. As well as many creative people. They may have a lot of product or design ideas (it's their job apparently). And the most dangerous situation is when your analysis contradicts with what they think.

E.g. manager tries to make a point and asks for an analysis. And it eventually ends up with a result opposite to what he or she expected. It's normal, and It happens very often - it's how hypotheses work. You make it, do an analysis and decide on whether you were right or not. But some people who don't trust data enough and can refuse to accept a result you give them.

Sometimes it happens with A/B tests. It could be hard to explain to people what statistical significance is and why we cannot roll out new feature if it performed "good" in test variant but you say "results are not significant". Or the worst case if it performed significantly bad but they expected it to be great. They are experts, they have a unique vision, it's impossible that feature is not working, eh? It happens, and you would be devastated if your scientifically-backed result is defeated by expertise with no real reason.

The problem intensifies when you make a mistake. It's hard to earn trust, and if you fail once, your data will be under the question mark for a long time, especially if it contradicts with expert's opinion. But people make mistakes, it's impossible to avoid them. The best way to deal with it is to accept your mistakes and understand and explain what went wrong. Well, anyway, biases are in the air, they are waiting for you around the corner.

Summary

There're a few key things I would emphasise:

Know your "audience". Make sure that your analysis/model/data matches with people capabilities to understand and use it.
Deliver+Control. It's always good to not only post a report/analysis/model but also make sure that your audience is using it correctly.
Democratising data is an opportunity, however, it works only if everyone has at least basic data literacy and statistical/math knowledge and understanding what is possible to do with their data and what is not.
Make any efforts to gain trust for your data, especially from upper management, and always accept your mistakes.
Learn and teach data storytelling.

Someone told me that a good blog post must contain a quote. So I picked up one but changed it a bit.

“Company shouldn't be afraid of their data. Data should be afraid of its company.” - Anonymous Data Scientist, original quote by Alan Moore, V for Vendetta

Reading

If you're interested more in topic of Data Culture and Data Literacy I would definitely recommend this book:

Creating a Data-Driven Organization

and also this article:
Are You Setting Your Data Scientists Up to Fail?

Thanks for reading! If you have any thoughts leave comments below or @ me on Twitter.

I ❤️ receiving feedback of any kind.

Also If you like this post you can share it and subscribe for updates. And... have a nice day!😉