Product and data teams struggle to work together because there's a tradeoff in data between flexibility, performance and cost-effectiveness.
Data and product teams always work smoothly; yeah, sure.
Why is it so hard for data and product teams to work together?
Their conversations go something like this:
Product team: We need to build this insights dashboard in our console and mobile app for customers to see the outcome of XYZ by the hour and be able to slice and dice by A, B, C, D, and E.
Data team: We only have that data monthly, and you can only slice and dice by C, D, or E. How important is this? How fast do you need to query it?
Product team: We need to be able to query super-fast (sub-second), and it is crucial to meeting our goal, so it is super important.
Data team: We can't do it. File a ticket with the requirements, and we can see where it falls in the priorities.
The product team is frustrated because they can't build the product they envisioned.
The data team is frustrated because they are constantly bombarded with high-priority tasks that are impossible to prioritize against each other.
What's the problem?
It is not that the product team is unreasonable or the data team is incapable. The product team is looking to build what's best for their customer. The data team seeks to inform decisions as effectively as possible across the organization.
To understand the root of the problem, we must dig deeper. There is a fundamental tradeoff in data between flexibility, performance, and cost-effectiveness.
Flexibility: refers to the different ways to slice and dice the data. What filters do you have? Can you see it by minute, hour, day, or month (its granularity), and how far back in time does it go?
Performance: refers primarily to the query speed. How fast is a given query?
Cost-effectiveness: refers to the total cost of implementing the desired outcome, with compute and storage being the primary factors driving the cost.
You can have two data characteristics above at the same time, but not all three.
You can pre-aggregate data which will give you performance and cost-effectiveness, but you will have tragic flexibility limitations.
You can beef up compute to query as fast as possible in order to keep the performance and flexibility, but it will come at a steep cost.
You can have all the flexibility in the world and be cost-effective, but it will likely get very slow queries, even on a small scale.
So what's the problem?
The root of the problem is not the tradeoff itself but who makes the tradeoff.
In the previous conversation, the data team made a tradeoff that sacrificed flexibility. At the time, it was the right tradeoff. They are also the team empowered and equipped to make such a tradeoff.
Unfortunately, that tradeoff did not work for what the product team needed. It would be surprising if it did, as it was made a long time ago with other requirements in mind.
Now imagine the product team has a huge budget. Their big budget can’t get them the performant and flexible product they want because it is blocked by the data team's ability to prioritize it. The data team's resources are very constrained, and the organization has many other priorities which make the product’s team ask fall below the line every quarter and therefore it never gets done.
Unlike the data team, the product team is not empowered or equipped to make this tradeoff.
The root of the problem is that the team that is closest to the customer is not empowered to make and execute the tradeoff.
The data team gets pulled in a million different directions and is not in a position to prioritize what's best for an individual product team.
What now?
The solution is for product teams, who are closer to the customer experience, to be empowered and equipped to make the tradeoff. How does an organization get there without each product development team having data engineering resources?
Our goal at Propel is to empower product development teams to build in-product analytics quickly and easily. We want product teams to be able to make the tradeoff they need to build their product. Propel does this by:
Simplifying data requirements. There’s no need to pre-aggregate, roll-up data, or create complicated indexes; just point us at your data.
Easily scaling compute. Product development teams can authorize applications with different levels of computing power with different levels of computing power that enable them to prioritize flexibility and performance.
Optimizing common queries. When a team builds a dashboard, there are a common set of queries that power it. Once developers know what those common queries are, they can optimize them for blazing-fast response times with just one click.
If you are interested in what we are doing, sign up for monthly product updates, follow us on Twitter @propeldatacloud, or join our waitlist to kick the tires on the product.
Top comments (0)