Setting and tracking appropriate target metrics are an important part of a product manager’s job. Goals must be defined, not just as inspiring vision statements, but also as quantifiable targets that can be objectively measured. Metrics can be deployed in different contexts and for purposes; however, useful metrics in one scenario can be misleading in regards to another. The problem comes when you’re not clear about what kind of metric you’re trying to set.
In this post I’ll be outlining six different types of metrics that you may track and what purpose each one serves. These aren’t the only types out there, but they are ones that can be easily misconstrued, and therefore require some differentiation and attention.
Success metrics: How do we know we did a good thing?
When considering your goals, objectives, and ultimate vision, can you put them into quantifiable terms and define your success metrics? Some examples: Increase watch time by 50 percent. Cap global warming at 1.5 degrees Celsius. Earn USD $100 million in revenue. A success metric should answer the question, “If we did all of the things in our roadmap, how do we know we did a good thing, rather than just wasting a lot of people’s time and money?” (See Metrics, Product Management, and You: Knowing you did a Good Thing for a deeper dive into this concept.) These metrics guide and focus the team and they should inform every major roadmap, feature, and strategy decision the team makes. With each step, you should ask: Does this help us meet our success metrics, or not?
However, success metrics tend to be long term. You might not know if you’ve actually met your goal until long after your work has been delivered. Per our global warming example: You can start reducing your greenhouse gas emissions today, but you won’t know if you’ve hit the 1.5 degree goal for years. This is actually a trailing indicator. It can’t tell you if you’re heading down the right path, because by the time you have your answer, it’s too late to change course. You need a different type of metric for that.
Leading indicators: Are we on the right track?
When determining your success metrics, it can be valuable to define a handful of leading indicators as well. As you pursue your goals, what signs might make you pause and reconsider your approach? What would hint that you’re on the right track and should double down?
Let’s say you are releasing an enterprise app internally at your company and you are aiming for 100 percent voluntary user adoption; however, you’re still rolling out to your first group of beta users and are nowhere near the capability set needed to satisfy the entire target user base. How would you know if you’re headed in the right direction? This might be as simple as a metric around user feedback, such as a net promoter score (NPS) or customer satisfaction ratings. Just make sure you remember that these are a supplement to your success metrics, not a replacement for them. An NPS of 100 among beta users does nothing for a product that doesn’t go to market.
Execution goals: Did we do what we said we were going to do?
Another approach that can act as a leading indicator is to track whether you’re actually executing on the strategy you committed to.
Let’s say you have a goal of improving your health and therefore set a success metric of lowering your resting heart rate. To achieve this goal, you decide to take the strategy of adding more cardio to your workout routines. You set a target of running three times a week. Whether or not you actually achieve this becomes your execution metric: are you executing on the strategy you decided on? If you fail to meet this metric, it might mean that you simply dropped the ball and need to refocus. But it could also be a signal that this strategy isn’t the best one for you. Maybe your knees start hurting, or you have trouble fitting a run into your day. Or maybe you just hate running and can’t get motivated. Rather than just double down, this is a chance to reconsider your strategy. A regular meditation practice or cutting back on caffeine could be approaches that fit your lifestyle better.
An execution metric can be a stand-in for a leading indicator (which can sometimes be difficult to identify or measure). It can also be an evaluation tool against your other metrics: If you don’t meet your success metric or leading indicators, is that because you weren’t executing your plan, or was your plan not effective? This determines your next course of action. The important thing here is not to mistake your execution goals for success metrics. If you run three times a week and your heart rate doesn’t go down, you will need to pivot regardless.
Decision informers: What choice should we make?
We’ve talked so far about big, strategy-driving metrics. But a product manager’s day-to-day responsibilities involve making all kinds of data-driven decisions. There are metrics you will gather to help you make specific decisions other than pivot or persevere. What version of a call to action should you use? A/B testing can help you decide. Can you improve simplicity and maintainability by removing some features? Check your analytics to see what is underutilized. Should we invest more in our Android or iOS apps? See what devices your users have.
As with the other types of metrics listed above, it’s good to set your target metrics ahead of time. For the example of your A/B test, you’ll want to have a cutoff for how long to run the test (e.g., For a specified number of days? Until you have a statistically significant amount of results?), as well as how big a performance gap is needed (e.g., Does a 1 percent improvement in performance merit a decision? 10 percent?). Otherwise, you risk making a decision prematurely—or not making one at all.
Monitoring metrics: the unknown unknowns
Not every analytics data point needs to tie back to a specific decision you are trying to make. There’s also value in just getting as much app usage data as you can so that you can observe unexpected patterns and trends. You might not realize there is even a decision to be made until you see your expectations about user flows wiped away by real life data. Especially early on in a product’s lifecycle, this type of data can be an invaluable source of truth about what happens when your product assumptions have their first brush with reality. These are the unknown unknowns—the things you don’t realize you should have had questions about. For example, one product I worked on had a search feature, and we assumed users would perform a search, review the results, and either take an action or close out of the app. When we looked at the analytics data, we saw that instead, the user flow typically had the user backing out through several screens and running additional searches—they wanted to compare different results, rather than perform an action on one. Once we learned this, we were able to redesign the interface to better accommodate this user journey. But if we had not had all these steps in the workflow tagged, and an analytics tool that pieced together the flow for us, we never would have thought to question our assumptions around the typical user flow.
In order to make these analytics valuable, and not just an overwhelming repository of data that will never get used, the best thing to do is to capture your assumptions ahead of time. Ask yourself: What is your expected user flow? What screen do you think will get the most hits? What’s your best guess at what your conversion funnel will look like? If you start from a place of uncovering your assumptions, you’ll be primed to notice trends that can be worth digging into.
Vanity metrics: Make us look good!
Product managers are often told that vanity metrics—those that look positive but don’t actually tell you anything meaningful—are a flat-out no-no. For example, cumulative time spent by all users on your website over time would be a vanity metric. Why? Because that number, by definition, will only ever go up; it’s additive. If yesterday the cumulative time spent was 10,000 hours, and today users only spent 1 hour on the site, your graph goes up from 10,000 to 10,001. This is a vanity metric because it looks good on paper if you’re not thinking about it too much. Look at that line, it’s trending up and up! But it doesn’t actually tell you anything meaningful. If, instead, you looked at total hours spent week over week, you could actually see more clearly if users are spending more or less time on your site over time. That is information that you can respond to.
However, I want to contest that vanity metrics are always bad. If you have something that looks impressive, use it! Put it in a powerpoint deck, in a press release, in your marketing materials. Getting people excited about your product is part of the job. The danger is in mistaking vanity metrics for leading indicators or success metrics. In other words, don’t believe your own hype!
As you can see, each of these types of metrics has its own value when used in the correct context. But trying to use the wrong metrics for the wrong reason—such as using vanity metrics as leading indicators, or execution goals as success metrics—can be detrimental to your product strategy. When you are clear about what kind of metrics you are using at what time, then you have the entire toolkit at your disposal to make the right decisions for your product.
Want more product management advice from experts in the field? Download the VMware Tanzu Labs Product Manager Playbook.
