DEV Community

Cover image for When Product Strategy Meets AI: What Happens to Metrics, Alignment and Decisions?
Romina Elena Mendez Escobar
Romina Elena Mendez Escobar

Posted on

When Product Strategy Meets AI: What Happens to Metrics, Alignment and Decisions?

#ai #product #data #management

1. Introduction

I have worked on application analytics projects, and there is one question that comes up again and again: what does it mean for our product to be successful?

It seems like a simple question, but agreeing on the answer can be much more difficult than it appears.

Marketing may focus on attracting new users and encouraging them to return. Product may prioritize feature adoption and user experience. Fraud and Security teams may focus on reducing risks and protecting the application. They all contribute to the success of the product, but each department looks at it from a different perspective and has its own priorities.

This diversity of objectives is also reflected in the questions that arise when analyzing data: What should we measure? Which indicators are relevant? How do we know if we are moving in the right direction? These questions are not always easy to answer when each team starts from a different definition of success.

Data governance can help establish common criteria for managing, interpreting, and using information. However, there is a question we need to address first: before agreeing on how to measure results, we need to agree on what we want to achieve as an organization.

How can we establish a shared direction that connects the priorities of different teams with a common objective, without losing sight of their specific needs?

In this article, I will explore how the North Star Framework can help answer that question, how to put it into practice through a workshop, and how generative artificial intelligence can support the process, from defining a primary metric to analyzing results and planning new initiatives.

2. When Measuring More Doesn't Mean Understanding Better

Today, we have tools that allow us to collect and analyze large volumes of information about how people use our products. We can measure downloads, active users, retention, conversions, or feature usage. However, having all these indicators does not necessarily mean we understand what is working or what we should improve.

The challenge is not only deciding which metrics to use, but understanding what they tell us about the product and how they relate to the objectives we want to achieve. An indicator may show positive results for one team without necessarily representing an improvement in the user experience or the value users receive.

For example, increasing the number of app downloads may seem like a good result. But what happens if those new users never return? Can we consider the product to be growing sustainably?

These questions lead us to distinguish between measuring product activity and measuring the value it actually generates.

The North Star Framework proposes addressing this challenge by establishing a shared direction: identifying a primary metric that represents the value users receive and connecting it to the factors that different teams can influence.

It is not about measuring less or replacing the indicators each department needs, but about giving them a common context that allows us to interpret results, connect decisions, and understand how each initiative contributes to the success of the product.

 3. What Is the North Star Framework?

The North Star Framework is a methodology that helps organizations identify what product success means and how to measure it, using the value users receive as a reference point.

Its starting point is the North Star Metric (NSM), a primary metric that aims to reflect the value a product delivers to its customers and its relationship with sustainable business growth.

However, the framework goes beyond selecting a single indicator. It also helps identify the factors that influence this metric, establish relationships between them, and guide the decisions and initiatives of different teams.

3.1 Origin and Evolution of the Concept

The concept of the North Star Metric became popular in the field of growth hacking, driven by Sean Ellis, as a way to identify the metric that best represents the core value a product offers to its users.

Later, Amplitude developed and promoted the North Star Framework, expanding the concept of a primary metric into a more structured methodology.

This approach incorporates what are known as input metrics, which help identify the factors that teams can influence to impact the North Star, as well as different models of value creation depending on the type of product and practices for connecting strategy with the decisions made by each department.

In this way, the North Star is no longer just a reference indicator but becomes the center of a framework that connects user value, product metrics, and business decisions.

3.2 What Does a North Star Metric Bring?

Defining a North Star Metric is not simply about choosing an indicator that represents product success. Its true value lies in helping us understand how the decisions and actions of different teams contribute to creating value for users and driving business growth.

This approach allows us to move from analyzing metrics in isolation to interpreting them as part of a shared strategy. Below, we can identify six key benefits of adopting a North Star Metric.

Ultimately, it is not about measuring less, but about measuring with a clear purpose. The North Star does not replace the indicators that each team needs to manage its activities, but rather provides a strategic context for understanding how they relate to one another and how they contribute to the success of the product.

3.3 North Star Metric vs. One Metric That Matters (OMTM)

Although both concepts aim to focus attention on a relevant metric, the main difference lies in their purpose and the time horizon they represent.

The North Star Metric (NSM) provides a long-term strategic direction. It aims to reflect the value the product delivers to its users and serves as a reference point for connecting the decisions of different teams with the overall business objectives.

On the other hand, the One Metric That Matters (OMTM) allows teams to focus their efforts on a specific priority during a defined period. Unlike the North Star, it does not aim to represent the overall success of the product, but rather to help a team address a specific challenge and evaluate the results of the actions implemented.

For an OMTM to be useful, it should meet three conditions:

  • Be specific: focus on a concrete, measurable outcome rather than a broad objective such as “improving retention.”
  • Have a defined time frame: establish a working period, such as a few weeks or a quarter, that allows progress to be evaluated.
  • Address a current priority: adapt to the product's needs and change when the expected results are achieved or new challenges arise.

Both metrics can coexist and complement each other. While the North Star establishes the strategic direction we aim to achieve, the OMTM helps identify where to focus efforts at a given moment.

In this way, an organization can maintain a shared long-term vision without losing the flexibility needed to respond to specific challenges and opportunities.

3.4 How Do We Know If We've Chosen a Good North Star Metric?

So far, we have explored what a North Star Metric represents and how it can help establish a shared direction. But this leads us to an important question: how do we know if the metric we have chosen truly reflects the success of our product?

There is no universal metric that works for every organization. The choice depends on the product, the needs of its users, and the business strategy.

Amplitude proposes a series of criteria that can help us evaluate potential metrics. Rather than treating them as a checklist of requirements, I find it interesting to use them as questions that encourage us to reflect on what we want to measure.

  • Are we measuring the value the customer receives? A good North Star should represent a real benefit for those who use the product, rather than simply the number of interactions they have within it.
  • Is it connected to our strategy? The metric should reflect what we aim to achieve as an organization and remain relevant beyond the priorities of a specific team.
  • Does it help us anticipate success? Not all indicators serve the same purpose. Metrics such as Monthly Recurring Revenue (MRR) or Average Revenue per User (ARPU) allow us to evaluate business results, but they do not necessarily help us understand whether we are creating the conditions for sustainable growth.
  • Can we influence its evolution? It is important that teams can identify actions capable of improving it. Measuring external factors over which we have no control may provide context, but it is unlikely to help guide our decisions.
  • Is it understandable across the entire organization? A North Star should be easy to explain without relying on complex technical definitions. If each team interprets it differently, it will be difficult to use it as a shared reference point.
  • Can we measure it reliably? We need a clear definition and data that allow us to track its evolution. However, there is something I consider important: we should not choose a metric simply because we currently have the data needed to calculate it. Sometimes, measuring what truly matters requires tracking new events, improving data quality, or developing additional analytical capabilities.

This does not mean that these indicators are useless. They can be essential for evaluating specific initiatives, but we need to question whether they are suitable for representing the product's primary objective.

Choosing a North Star Metric should not be a competition between departmental indicators, but an opportunity to build a shared vision of what success means.

And this brings us to a question worth exploring separately: when does a metric that appears positive become a vanity metric?

3.5 Vanity Metrics: What Are We Trying to Demonstrate?

In many organizations, each team uses indicators to evaluate and demonstrate the results of its work. Marketing may highlight user growth, Product may focus on feature adoption, and Sales may emphasize increased conversions. All these results can be positive, but does that mean the product as a whole is successful?

This is where the risk of so-called vanity metrics arises: indicators that show favorable results but, when analyzed in isolation, do not necessarily reflect the value users receive or help us make better decisions.

A common example is the number of Daily Active Users (DAU). Having more and more people open an application may seem like a good sign, but does it mean they are actually getting value from it?

The answer depends on the context. For some applications, frequency of use may be essential, while for others, what matters is that users complete a task or fulfill a specific need. Even within the same product, the relevance of a metric can change depending on its stage of maturity. A newly launched application may prioritize user acquisition, while a more established one may focus on retention.

That is why a metric is not inherently a vanity metric, but becomes one depending on what it represents, how we interpret it, and what we use it for.

The challenge is not to stop measuring the performance of each team, but to avoid confusing positive results from individual indicators with the overall success of the product. The North Star Framework helps us connect these different perspectives with a shared objective.

4. From Framework to Practice: How to Organize a Workshop

So far, we have explored what the North Star Framework is and which criteria can help us identify a metric that represents the value our product generates. But there is an important question: how do we define that metric when each team has its own priorities and interpretations of success?

One way to address this challenge is through a collaborative workshop, as suggested by Amplitude in its documentation, that brings together different perspectives and helps transform ideas, data, and hypotheses into a shared direction.

To illustrate how I would put this process into practice, I designed a hypothetical exercise based on a language learning application like Duolingo. It does not represent the company's internal processes, but will serve as an example throughout the different stages of the workshop.

4.1 Who Participates

The first step is to identify the stakeholders who should participate in the workshop. To build a shared vision, we need to bring together different perspectives on the product, including areas such as Marketing, Product, Business, Data, Engineering, and, depending on the context, Fraud or Security.

We can use tools such as the Power–Interest Matrix to identify relevant stakeholders and understand their level of influence and interest. However, we should not select participants solely based on their hierarchical position, but also on the knowledge and experience they can contribute.

The goal is not for everyone to share the same priorities, but for their different perspectives to contribute to defining a common direction.

And these differences are reflected not only in each team's objectives, but also in the beliefs and assumptions that each person brings to the workshop.

4.2 The Beliefs and Assumptions We Bring to the Workshop

Once we have identified the stakeholders, there is another aspect we need to consider: each person comes to the workshop with their own ideas about what makes the product successful. Some are supported by data, while others come from experience, intuition, or the priorities of each team.

Amplitude's North Star Playbook identifies different types of beliefs that can influence these conversations:

  • 〰️ 👥 Customers: what we believe our users need and value.
  • 〰️ 🔗 Causality: which actions we think lead to certain outcomes.
  • 〰️ 📈 Market: how we believe the market will evolve and what opportunities may arise.
  • 〰️ 🤖 Technology: which trends could transform our product.
  • 〰️ 🏁 Competition: what we think other players are doing or planning.
  • 〰️ 💎 Value Proposition: what we believe differentiates our product.

The problem is not having these beliefs, but confusing them with facts that we have not yet validated.

For example, in our hypothetical language learning application, someone might say: “Users come back because they want to maintain their learning streaks.” This may sound reasonable, but do we have data to support it, or are we starting from an assumption?

Making these differences visible allows us to identify what we know, what we need to validate, and which hypotheses should guide our analysis.

And this is especially important in a workshop involving different departments: our goal is not to let the opinion of the person with the most influence prevail, but to build decisions based on collective knowledge and the available evidence.

4.3 Before the Workshop: Creating the Right Environment

Once we have selected the participants and identified the different perspectives they can contribute, we need to prepare an environment that encourages participation and the exchange of ideas.

In a workshop where we aim to define a shared direction, the way the conversation unfolds can be just as influential as the ideas that emerge from it. That is why it is important to establish certain conditions from the beginning:

  • 〰️ 🤝 Build trust: create a space where people can share ideas, ask questions, and challenge assumptions without fear of being judged.
  • 〰️ 📏 Establish clear rules: agree on the time allocated to each activity, respect speaking turns, and encourage discussions focused on ideas rather than individuals.
  • 〰️ 🏢 Balance hierarchies: prevent the opinions of those in positions of greater responsibility from influencing everyone else. One alternative is to collect ideas individually before opening the discussion.
  • 〰️ 🔕 Avoid interruptions: set aside dedicated time and space to maintain focus and continuity throughout the activities.

The goal is not to eliminate differences of opinion, but to allow all perspectives to be expressed before reaching a conclusion.

With these conditions in place, we can begin working on the activities that will help us transform different ideas into a proposed North Star and its input metrics.

4.4 During the Workshop: From Ideas to a North Star Metric

Once we have identified the participants, recognized the different beliefs, and established the conditions for working together, we can begin the workshop.

To illustrate the process, I will use a hypothetical example based on a language learning application like Duolingo. It is not intended to represent how the company works internally, but rather to help us visualize how we can move from different perspectives to a North Star Metric and its input metrics.

We will organize the workshop into three stages: divergence, convergence, and hypothesis definition.

4.4.1. Divergence: Generating Ideas

The first objective is to gather different perspectives without trying to reach a conclusion immediately.

We can start with a question that encourages reflection:

What do we believe drives the success of our application?

And complement it with some additional questions:

  • 〰️ Why do we believe users continue using the application?
  • 〰️ At what point do they feel they are actually learning?
  • 〰️ What factors might cause them to abandon it?
  • 〰️ What would motivate them to return after a break?

To carry out this activity, we can use the 3-12-3 Brainstorming technique: three minutes to write down ideas individually, twelve minutes to share and develop them in pairs, and three minutes to present the main proposals to the group.

An interesting practice is to use two different colors of sticky notes to distinguish ideas supported by data from those that are still assumptions.

At this stage, we are not trying to determine who is right, but rather to gather different perspectives and prevent some ideas from being dismissed before they have been heard.

4.4.2. Convergence: Grouping and Prioritizing

Once we have collected the ideas, the next step is to organize them to identify what they have in common and how they relate to one another. At this stage, we are no longer trying to generate new proposals, but rather to find patterns that help us understand which factors could contribute to the product's success.

To do this, we can use Affinity Mapping, a technique that involves grouping ideas based on their themes or similarities.

Returning to our language learning application example, we could identify different groups related to motivation and learning habits, how easy it is to get started, the ability to resume lessons after a break, or users' perception of their own progress.

As we organize these ideas, we can also begin to distinguish between the outcomes we want to achieve and the actions or factors that could help us achieve them. For example, encouraging users to return daily may be a desired outcome, while timely reminders or features related to learning streaks could be factors that influence this behavior.

However, not all ideas will be equally relevant or supported by the same level of evidence. To identify which ones participants consider most important, we can use Dot Voting, a technique in which each person receives a limited number of votes to select the proposals they consider priorities.

Voting does not determine which ideas are correct, but rather which ones deserve greater attention. If a proposal receives many votes but is still based on assumptions, we have an interesting hypothesis that needs to be validated before making decisions.

During this process, questions or topics may also arise that we cannot resolve at that moment. To prevent the conversation from losing focus, we can use a Parking Lot to record these issues and revisit them later.

In this way, we move from a collection of individual ideas to an initial organization of potential outcomes, influencing factors, and hypotheses that will serve as a starting point for defining a North Star Metric and its input metrics.

4.4.3. Defining the Hypothesis: Connecting the Metrics

With the ideas grouped and prioritized, we can begin to build a hypothesis about which factors contribute to the product's success.

The goal is to identify a North Star Metric that represents the value we want to create and connect it to a limited set of input metrics that we can influence.

To express this relationship, we can use a phrase such as:

“If we do all of this right, then…”

This exercise encourages us to explain why we believe that improving certain factors will contribute to the outcome we want to achieve.

From there, we can identify opportunities for improvement, propose potential interventions, and define how we would measure their results.

It is important to remember that not every idea automatically becomes a metric, and not every relationship we propose has been proven. The outcome of the workshop is a shared hypothesis that we will later need to validate with data.

4.4.4. Documenting: The Workshop's Memory

At the end of the workshop, it is important to document not only the decisions made, but also how we arrived at them: the ideas proposed, the groups identified, the votes, the hypotheses that still need to be validated, and the reasons why certain alternatives were discarded.

This documentation allows us to maintain traceability of decisions and revisit the reasoning that led us to define our North Star and its input metrics.

But there is another aspect I consider important: metrics are defined within a specific context, and that context can change. That is why it is also useful to record the product's stage of maturity, the features it offered, who its users were, and what the business priorities were at that time.

Consider, for example, how Duolingo has evolved: it went from offering language courses to incorporating new learning experiences, other disciplines, and AI powered features. These changes can influence how people use the product and the value they expect to receive.

The same can happen due to external factors. The emergence of new technologies, changes in the market, or evolving user needs may mean that a metric that was once relevant needs to be reviewed.

Documenting the context allows us to understand why we made certain decisions and recognize when it is necessary to question them again.

4.4.5. After the Workshop: Validating and Reviewing

The workshop allows us to build a shared vision, but the metrics and relationships we have identified still need to be validated with data. Reaching an agreement does not necessarily mean that our hypotheses are correct.

Validating the Hypotheses

Once the session is over, we can transform ideas and assumptions into concrete hypotheses that allow us to analyze which factors influence our North Star.

This involves reviewing the available data, identifying what information is missing, and defining how we will test whether the proposed relationships actually hold true.

Here, generative AI can help us organize workshop notes, structure hypotheses, and prepare questions for analysis. However, validation requires evidence and the judgment of the teams, not just an interpretation generated by AI.

Reviewing Periodically

Organizations evolve, products incorporate new features, and user needs change. That is why a North Star should not be considered a permanent decision.

It is advisable to establish periodic reviews to evaluate whether the metric continues to represent the value the product generates and whether the defined inputs remain relevant.

The goal is not to change metrics constantly, but to ensure that we continue measuring what truly matters.

5. A Real Case to Understand Input Metrics: Duolingo's Growth Model

In the previous section, we explored how we can use a workshop to define our North Star Metric and identify the factors we believe influence it. But how can we connect a primary metric to more specific indicators that help us understand what drives its results?

To better understand this, I will use the growth model that Duolingo published in 2023 as a reference. It is an interesting example of how a company can leverage its product data to understand user behavior and guide its decisions.

Faced with stagnating growth in Daily Active Users (DAU), Duolingo's Data Science team developed a Growth Model based on a Markov model to analyze how users transitioned between different activity states.

Rather than looking only at the total number of active users, the model distinguishes seven states based on user behavior:

  1. New Users: users who open the application for the first time.
  2. Current Users: users who are active today and were also active during the past week.
  3. Reactivated Users: users who return after more than a week of inactivity, but within the past month.
  4. Resurrected Users: users who return after more than 30 days of inactivity.
  5. At-risk WAUs (Weekly Active Users): users who were active during the past week, but not today.
  6. At-risk MAUs (Monthly Active Users): users who were active during the past month, but not during the past week.
  7. Dormant Users: users who have not used the application for at least 30 days.

Illustration of the Duolingo Growth Model: Technical Details

What makes this model interesting is that it does not simply classify users, but also allows us to analyze how they move from one state to another and which factors may contribute to their continued use of the product, their return after a break, or their decision to stop using it.

To study these transitions, Duolingo uses different retention metrics, including four worth highlighting:

  • 〰️ CURR (Current User Retention Rate): measures the retention of regular users.
  • 〰️ NURR (New User Retention Rate): analyzes the retention of new users.
  • 〰️ RURR (Reactivated User Retention Rate): measures the retention of users who return after a relatively short break.
  • 〰️ SURR (Resurrected User Retention Rate): analyzes the retention of users who return after an extended period of inactivity.

Using these metrics, Duolingo ran simulations to understand which ones could have the greatest impact on DAU. In its analysis, the team identified that improving CURR offered a particularly significant opportunity and used A/B experiments to evaluate strategies aimed at improving retention.

What I find interesting about this example is how a general metric can be broken down into more specific indicators that help identify where to take action.

We can also apply this logic to the North Star Framework. We do not always need to create new metrics from scratch. Many organizations already have indicators related to acquisition, activity, retention, or conversion. The challenge may lie in selecting the most relevant ones, understanding how they relate to each other, and connecting them to the value we want to create for our users.

In other cases, we will need to introduce new ways of measuring or review existing metrics as our product evolves.

It is important to clarify that Duolingo's Growth Model is a product analytics model and not necessarily its official North Star Framework. However, it helps illustrate how we can identify factors that influence a primary metric and use them to guide our decisions.

But selecting a limited set of input metrics does not mean that other indicators are no longer important.

An application still needs to analyze other dimensions, such as user acquisition, feature adoption, user experience, retention, and product quality.

These complementary metrics provide context to help us interpret results, identify problems, and understand what is happening beyond our North Star.

6. From the North Star to Contextual Metrics

So far, we have explored how a North Star Metric can help us establish a shared direction and how its input metrics allow us to identify the factors that contribute to achieving it.

But the success of an application cannot be understood through a single indicator or a limited set of metrics.

Teams need to analyze different aspects of the product: how users arrive, what happens during their first interactions, which features they use, why they return, and whether they encounter technical issues that affect their experience.

These metrics provide context and allow us to interpret results from different perspectives. For example, a decline in retention could be related to performance issues, difficulties during onboarding, or changes in the behavior of certain user segments.

The following infographic brings together some of the KPIs we can use to analyze different dimensions of an application, from acquisition and activation to engagement, retention, and product quality.

The North Star helps us maintain a strategic direction, while contextual metrics allow us to understand what is happening within the product and which aspects we need to investigate.

It is not about constantly analyzing every indicator, but rather selecting those that provide relevant information to interpret results and make better decisions.

7. How AI Can Support the North Star Framework

The North Star Framework does not end when we define a metric. It is an ongoing process that requires analyzing results, testing hypotheses, communicating decisions, and reviewing whether we are still creating value for our users.

Throughout this process, artificial intelligence can help us work with information, explore different perspectives, and transform data into useful knowledge to support decision-making.

However, it is important to distinguish between generative AI and other artificial intelligence techniques. While machine learning can be used to detect anomalies, identify patterns, or build predictive models, generative AI offers other capabilities, such as summarizing information, facilitating its interpretation, and allowing people to interact with data through natural language.

The following infographic summarizes five stages in which AI can support the application of the framework, while keeping validation and final decisions in human hands.

🧭 1. Define the North Star

During workshops, generative AI can serve as a supporting tool to organize ideas and broaden participants' perspectives.

For example, we can provide it with information about the product, interviews, surveys, user feedback, and internal documentation to identify recurring themes, compare opinions, and detect potential contradictions.

It can also help us transform workshop ideas into metric proposals and evaluate them using the framework's criteria.

What makes this interesting is not having AI choose our North Star, but using it to help us ask better questions and explore alternatives we may not have considered.

🔬 2. Validate the Metric and Identify Its Inputs

Once we have defined the candidate metrics, we need to verify whether they represent the value we want to create and whether we can measure them reliably.

Generative AI can help us review definitions, identify ambiguities, and translate business questions into SQL queries that must subsequently undergo technical validation.

We can also complement this work with machine learning techniques to explore relationships between variables, identify patterns, and estimate which indicators are most strongly associated with changes in our North Star.

However, finding a correlation does not mean proving causation. The relationships identified need to be tested through appropriate analysis and, when relevant, experimentation.

🗣️ 3. Explain Results to Different Teams

One of the challenges of working with metrics is ensuring that people with different responsibilities and levels of technical knowledge can understand and interpret them.

Here, generative AI can help transform complex analyses into explanations tailored to each audience, connecting changes in the North Star with input metrics and contextual indicators.

It also allows people to explore data by asking questions in natural language, without requiring everyone to know how to write SQL queries.

The goal is not for every team to analyze the same indicators, but for them to understand how their metrics relate to the product's results.

📡 4. Monitor and Understand Changes

Once the metrics have been implemented, we need to track their evolution and understand what happens when results change.

We can use analytics tools to detect anomalies and combine their outputs with generative AI to prepare reports, summarize trends, and identify potential areas for investigation.

But there is one question I consider particularly important: are we observing a real change in user behavior or a problem with our data?

A sudden decline in activity may be caused by a change in the product, but it could also result from instrumentation errors, events that are no longer being recorded, or issues in data pipelines.

That is why monitoring should not be limited to business indicators. We also need to consider data quality and technical changes that may affect how results are interpreted.

AI can help us connect this information and suggest possible explanations, but these hypotheses must be verified before being presented as conclusions.

🔁 5. Review and Plan New Initiatives

Finally, the results of our analysis should help us decide what to do next.

Generative AI can support the formulation of hypotheses, the preparation of improvement proposals, the design of experiments, and the documentation of lessons learned.

It can also help us explore scenarios, such as analyzing what might happen if we improve retention or reduce difficulties during onboarding.

These estimates can be useful for prioritizing initiatives, provided they are based on appropriate models and subsequently compared with actual results.

AI Supports the Process, but Does Not Replace Decisions
There is one principle that applies across all these stages: AI does not replace product knowledge, data quality, or the judgment of teams.

If each department uses different definitions for the same metric, introducing AI will not automatically resolve that lack of agreement. It could even amplify inconsistencies and generate incorrect interpretations.

Similarly, optimizing an indicator without understanding what it represents may lead us to improve the numbers without creating more value for our users.

The true potential of combining AI with the North Star Framework lies in helping organizations connect data, knowledge, and decisions while maintaining a shared direction without losing sight of the value they want to create.

8. Conclusions: Beyond Metrics and Technology

Throughout this article, we have explored how the North Star Framework can help us establish a shared direction, connect product metrics with the value users receive, and transform different perspectives into more aligned decisions. However, while defining a good metric is an important step, it does not guarantee that an organization will work toward the same goal, especially when team priorities, incentives, and dynamics are not aligned.

8.1 When Individual Goals Do Not Guarantee Collective Success

Metrics allow us to evaluate results, but they can also influence how people work and make decisions. When a team's performance depends on certain indicators, it is natural for them to focus their efforts on improving those metrics, even if doing so does not always contribute directly to the overall objective.

To illustrate this, I want to revisit an example I once heard involving football, where a striker can be evaluated by the number of goals they score and a goalkeeper by the goals they prevent. Let's imagine three scenarios:

  • 3 to 4: the striker scores three goals and achieves an excellent individual result, but the team loses.
  • 0 to 0: the goalkeeper keeps a clean sheet and achieves their objective, but the team does not win.
  • 2 to 1: the striker scores fewer goals and the goalkeeper concedes one, but the team wins.

If we look only at individual metrics, we might conclude that the striker performed better in the first scenario and the goalkeeper in the second. However, when we consider the collective result, the interpretation changes because, as a team, our goal is to win the match.

This does not mean that the striker's goals or the goalkeeper's saves are no longer relevant, but rather that we need to understand how they contribute to the shared objective. Something similar happens in organizations, where Marketing may achieve its acquisition targets, Product may improve feature adoption, and other departments may meet their own KPIs, without those improvements necessarily translating into a better user experience.

The challenge, therefore, is not to eliminate individual metrics, but to find a balance that allows us to evaluate each team's responsibilities without losing sight of the outcome we want to achieve as an organization.

8.2 People Are Also Part of the Equation

This reflection leads us to consider another dimension that is often left out of measurement models: the human and organizational context in which objectives are defined and results are interpreted.

Organizations are made up of people with different experiences, motivations, responsibilities, and ways of understanding success. That is why relationships between teams, trust, communication, and organizational culture also influence the decisions being made and how data is used.

Furthermore, when we talk about building a product, we usually think primarily of Product, Technology, or Marketing teams, but its success is also influenced by other departments, such as Human Resources, Administration, or those responsible for organizational management. Even if they are not directly involved in developing a feature, their decisions can influence working conditions, motivation, and the ability of those teams to collaborate.

For example, a performance evaluation system that rewards only the achievement of individual goals may end up encouraging behaviors that favor one department's results while making collaboration with other teams more difficult. In this context, metrics can become a way to demonstrate performance rather than helping us understand whether we are creating real value, bringing us back to the risk of vanity metrics.

That is why aligning metrics also means understanding the incentives and dynamics of the people who work with them. We can define a shared North Star, but if evaluation systems, priorities, or working conditions push teams in different directions, it will be difficult for that common objective to translate into consistent decisions and actions.

8.3 AI Can Help Us Think, but Not Decide for Us

This organizational complexity also brings a new opportunity: artificial intelligence, which can help us organize information, identify patterns, test hypotheses, and make data more accessible. However, its usefulness is not limited to analyzing metrics. It can also extend to the way we make decisions.

During the ideation stages, for example, generative AI can help us review participants' proposals, identify potential contradictions, or highlight perspectives that were not considered. This can be particularly useful when hierarchical differences, individual interests, or shared assumptions influence the conversation, as it allows us to introduce new questions and challenge some of the conclusions we reach.

Nevertheless, we must remember that AI can also reproduce biases present in the information it receives and that its interpretations are not necessarily correct. That is why its role should be to expand our capacity for analysis and reflection without replacing product knowledge, data quality, or human judgment.

Similarly, if each department uses different definitions for the same metric, introducing AI will not automatically resolve that lack of agreement and could even amplify inconsistencies. And if we focus solely on optimizing indicators without understanding what they represent, we may end up improving dashboard results without actually improving the experience of our users.

A Final Reflection

After exploring these concepts, I believe the greatest challenge is not necessarily finding the perfect metric, but building an organization capable of agreeing on what success means, identifying the factors that contribute to achieving it, and reviewing its decisions as the product, people, and their context evolve.

The North Star Framework can provide a structure to guide this process, and AI can help us analyze information and challenge our hypotheses, but neither of these tools replaces the need to build agreements, establish trust, and understand how different responsibilities contribute to a common purpose.

Because, ultimately, it is not just about aligning metrics, but about aligning people, decisions, and objectives around the value we want to create.

9. References

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.