TL;DR: Product managers working on software-based enterprise products face unique challenges in building product roadmaps oriented around measurable outcomes. This post aims to surface some of those challenges and suggest ideas for adapting to those challenges.
Current Landscape
Many modern software development methodologies prescribe:
-
balanced teams comprised of one or two product managers, “two pizzas” of engineers, and one or two designers
-
responsible for the design, development, testing, and delivery of a software-based solution to a fairly narrowly bounded scope of target user problems
-
optimized for rapid, incremental delivery to enable short feedback loops with their target market.
A company aimed at selling software-based products to an enterprise market must often provide an offering:
-
that solves a broad range of problems for a broad range of target users
-
that composes an experience from multiple disparate component solutions in a holistic manner, while imposing little burden on the end user
-
that is priced in a simple manner that does not require the buyer or customer decision maker to reason about consumption of sub-components of the overall offering
-
that is released at a cadence matching the pace of enterprise consumption, which is often very slow due to factors that are common to large enterprises, such as time-consuming internal compliance requirements and a generally conservative approach to consuming newly-released versions of products.
These realities create a unique set of challenges for product managers on modern software development teams who are contributing to a larger enterprise product offering which consists of the solution(s) provided by their team, especially when the product manager wishes to set their team’s direction based on measurable objectives. That desire to set direction based on measurable objectives is fairly common, especially with the popularity of frameworks such as Objectives and Key Results (OKRs) as a means of creating focus and alignment within organizations and teams.
It is worth noting that much of this discussion is applicable regardless of whether the product is customer managed (i.e. on-premises) or vendor managed (i.e. a cloud service like Salesforce.com or AWS). The features of the landscape described above still apply. There are some notable differences, however. Vendors that manage the software on behalf of an enterprise have the freedom (but also the responsibility) to update and operate the service at their own pace. This avoids bottlenecks that would otherwise be present in a customer-managed environment. Another notable difference is that a vendor managing their own service has greater control over how they instrument their services to give themselves immediate access to measurements of key objectives. These differences allow service providers to speed up certain parts of the feedback loop with customers. But this scenario does not circumvent the reality that enterprises will still be slow to actually consume and generate feedback on new features.
Measuring Measurable Objectives
To understand this unique challenge, it’s important to first reflect on what makes a measurable objective good. What qualities of a measurable objective might one measure to assess whether the objective is a good one?
-
Time to measure. How long does it take until it is possible to make a measurement? Sooner is better. “Engineers deliver feature X” can likely be measured in days. “Customer X renews their subscription at $Y” may not be measurable for several years, until the customer’s current subscription is set to expire and renewal discussions begin.
-
Ease of measurement. How much work does it take to actually measure progress toward an objective? Easier is better. “N customers download the version of the product that includes some capability” is often easier to measure than “N customers have not only downloaded the version of the product that includes some capability, but they are actively using the capability, and their usage of this capability is leading to increased satisfaction with the product.”
-
Correlation with a team’s output. To what degree can we confidently conclude that the value of the measurement actually has anything to do with what the team is doing? Higher correlation is better. “N customers actively use the new capability our team released” is much more highly correlated to the team’s work than “X% of customers renew their subscriptions.”
-
Correlation with the company’s success. To what degree does meeting the objective actually help the company? Higher correlation is better. “Company’s revenue increases X%” is obviously very highly correlated with the company’s success; “N organizations can successfully use the free version of our software in production” may make many users very happy but may in fact be anti-correlated with the success of the company.
-
Correlation with the users’ success. To what degree does meeting this objective actually deliver value to the users? Higher correlation is better. “N organizations can successfully use the free version of our software in production” is very highly correlated to delivering user value, while “We are able to create a new product SKU for an existing capability users are using, extracting greater revenue, with 0 impact on customers’ current and projected adoption of the product” is anti-correlated with user value.
-
Objectivity. To what degree can this be measured based on impartial observations of data rather than canvassing for opinions? Generally, more objective is better. “X% of our company’s operations team who manage our hosted offering say they like the new workflow” is less objective than “the operations team spends X% less time performing maintenance on the product by using the new workflow.”
The challenge for product managers is to define measurable objectives that are good across all of these dimensions, while these dimensions are often at tension with one another.
Visualizing the Tension
One way to visualize an evaluation of something along multiple, potentially competing dimensions is a radar chart or spider chart. It’s not uncommon to see this in the world of collegiate and professional sports. Here are radar charts for two quarterbacks in the 2005 Draft from the National Football League in the US:
One key metric in these charts is body weight. You might notice that Alex Smith is relatively very light (31st percentile amongst quarterbacks in terms of body weight) compared to Charlie Frye (61st percentile), which may make Alex easy to tackle or knock down by opposing defensive players: not good. And yet Alex performed better in the 40 Yard Dash than Charlie, meaning he’s a faster runner with likely greater acceleration. It’s reasonable to infer that body weight and speed are two measurables that are in tension with one another; it’s hard to be big and heavy but also fast and agile.
Similarly, it’s reasonable to assume tension between the various qualities of any good measurable objective. Given that enterprise product offerings are typically a holistic composition of the components and solutions produced by many small product teams, a measure that is highly correlated to an individual team’s output is likely very weakly correlated with the company’s overall success. Without a sophisticated telemetry system, understanding the usage and impact of output in an objective manner is hard, because it requires asking users to retrieve it, and it’s easier for them to give you their opinion than it is for them to extract and send you the data. In such a scenario, objectivity and ease of measurement can easily be in tension with one another.
What does it look like to visualize this tension for product objectives? Let’s take a look at an example scenario, define two possible objectives a product team may aim for given the scenario, and define a way to numerically score the objectives against the various qualities like “time to measure,” so that we can visualize them.
In this scenario, we observe that Amazon Web Services (AWS) is the most popular public cloud offering amongst enterprises in the US. Suppose we are on a team building a part of our company’s product offering, which is some software component that relies on some external service that manages credentials. We know that secure credential management is very important for enterprises. We have received explicit requests from five customers to enable our component to natively integrate with AWS Key Management Service (KMS) for credential management. If we want to define a measurable objective to align our team’s efforts in this scenario, to clarify the purpose of our work, and to give ourselves a clear way to measure success, the following are two possible objectives we could define:
-
After we ship the capability that those customers have asked for, those 10 customers confirm that they plan to use it in the next three months.
-
Revenue from customers using our product by deploying it to AWS increases 40% in the next year.
For the sake of simplicity, let’s define a 0-to-3 point scale for each attribute:
-
Time to measure
-
at least one year
-
one quarter to one year
-
one week to one quarter
-
less than one week
-
-
Ease of measurement
-
requires continuous manual data retrieval by individuals outside the team
-
requires one-time manual data retrieval by individuals outside the team
-
requires one-time manual data retrieval, can be done by the team itself
-
the measurements are automatically delivered to the team
-
-
Correlation with a team’s output
-
anti-correlated: work by our team will correspond to negative change in the objective
-
uncorrelated: work by our team doesn’t correspond to change in the objective
-
slightly correlated: work by our team likely corresponds to some positive change in the objective
-
highly correlated: work by our team highly likely corresponds to significant positive change in the objective
-
-
Correlation with the company’s success:
-
anti-correlated: a positive change in this objective results in a negative impact on company success
-
uncorrelated: a change in this objective has little impact on changing company success
-
slightly correlated: a positive change to this objective likely has some correspondence with a positive change in the company’s success
-
highly correlated: the company’s success is directly and highly correlated with positive movement towards this objective
-
-
Correlation with the users’ success:
-
anti-correlated
-
uncorrelated
-
slightly correlated
-
highly correlated
-
-
Objectivity:
-
the subjective opinion of internal stakeholders
-
the subjective opinion of a small sample of end users
-
the subjective opinion of a large sample of external stakeholders and end users
-
the objective measurement of some phenomena independent of any opinions
-
Based on those scales, radar charts for the two proposed objectives would look like this:
Clearly, neither objective is perfectly ideal, and the two objectives are good and bad in different ways.
Adapting to the Challenge
Given the current landscape around developing software products for an enterprise market, what can product managers do about the above challenge?
First and foremost, make the tension explicit. Be honest with yourself and your team what tradeoffs you’re making when defining your team goals. Be honest that there is a tradeoff, and make choices intentionally. Avoid the trap of jumping straight to picking objectives that are easier to measure without reflecting on the loss of choosing objectives that are more important to measure. If your team defines high level goals for itself on a quarterly, semi-annual, or annual basis, consider brainstorming multiple goals and depicting them on radar charts to make it easy for the team to reflect on the tradeoffs it is making as a group.
Secondly, consider diversifying your objective portfolio. That is, choose a mix of objectives that are good in different ways. Have some that are very concrete and can give the team a clear sense of accomplishment, and others that tie more closely to the larger company mission and give the team a sense of greater purpose even if it can be harder to truly see how the team’s accomplishments impact the larger mission.
Overcoming the Challenge
Given the current landscape of how software-based products tend to be developed on small teams oriented around rapid iteration, and how enterprises tend to consume software-based products at a relatively slow, conservative pace, it seems that a product manager can only hope to adapt to the challenge. But the landscape is shifting, and it may eventually be possible to fully overcome the challenge.
Enterprises are facing intense competitive pressure to bring new solutions to market faster and faster, and the solutions they offer to their customers must increasingly be delivered in digital form. They are being disrupted by competitors with a more modern approach to software, who are able to consume technology and build their own digital solutions with high velocity. Enterprises that are able to leverage the latest software-based products will have an advantage over slower moving enterprises, so we would expect to see shorter feedback loops for product managers measuring objectives related to the software-based products they’re delivering.
With the momentum of open source projects such as Kubernetes and the tendency of solutions to move towards commoditization and standardization, together with the increasing push for all organizations (including enterprises) to grow their technological competence, we may see a shift away from holistic, highly integrated, complex product offerings from vendors, and towards a rich ecosystem of narrowly scoped solutions conforming to open standards consumed by enterprises. In this world, the burden of composition and integration of these components shifts from vendors to consumers. This could lead to a decrease in the layers of abstraction between the output of a team and the experience of an end user, and a decrease in the layers of human interaction between the team producing the output and the buyer who makes the purchasing decision.
Lastly, with the growth of the SaaS and public cloud consumption model, more product managers may be able to enjoy greater control over how users consume their solution. And they can gain greater control over how they instrument their services to give themselves immediate access to measurements of key objectives.
But, regardless of whether these trends play out or whether they have the expected impact on product management, product managers will always have to balance the art of empathizing with their human stakeholders, anticipating the behaviour of markets, and intuiting the potential of new technologies with the science of measuring, synthesizing, learning, and constructing increasingly efficient feedback loops when it comes to decision making and setting product direction.
