This post was co-written by Moni Gmitro, Caleb Singer, Karl Meihofer, Andrew Woods.
Organizations are in the business of delivering with software, however, the new battleground of product differentiation no longer takes place on the factory floor. Nowadays, distinction occurs with the software supply chain in the form of rapidly delivered new features. These software supply chains require high investments and a complex system of technology, processes, and practices with the goal of turning silicon ones and zeros into new product features that users value and love.
Measuring software delivery is critical for the growth of businesses. In fact, it’s so important that the site reliability engineering (SRE) team at Google spent six years identifying the right metrics for measuring their software delivery process. Choosing which metrics are relevant is already challenging, but when it comes to telling the story behind the metrics, the challenge is palpable. Even when the right metrics are tracked, that does not mean interpretation of the data leads to accurate conclusions. This results in decisions that don’t enable healthy software delivery. Best case scenario: the organization may choose to do nothing about it. Worst case scenario: the story is optimized for actions that don’t drive towards outcomes. This raises the question of, “Are there ways to measure these software systems and can the results guide improvements to achieve better business outcomes and a competitive edge?”
Adopting a storytelling approach to software delivery performance that fosters better outcomes is crucial in helping teams fully grasp the context of software delivery metrics and make effective decisions.
What is software delivery measurement and why does it matter?
Software delivery measurement aims to improve the way software products and services are built. There are many frameworks and approaches—take for example, metrics gathered by the DevOps Research and Assessment (DORA) program and the Five S’s of Shipping Software. While these frameworks are a great starting point, interpreting the metrics they generate is a challenge that often leads to pitfalls or sub-optimization of the framework.
Here are some common pitfalls of misinterpreting software delivery metrics:
- Optimizing for one metric at the expense of another – Even though metrics represent distinct parts of the software supply delivery process, they are interdependent and need to be evaluated as a whole. Trying to game the system by only focusing on one metric often makes things worse. For example, in trying to increase deployment speed, we might have the undesirable side effect of increased change failure rate.
- Local optimizations – Attempting to apply metric optimizations in areas that are not true bottlenecks in the system can result in little or no improvement to the overall system. For example, trying to quicken build and release times by doubling compute resources in a system that still has a manual step that has to be fulfilled before the build goes to production.
- Misreading data narratives – It's crucial to ensure a logical connection between the events or stories and the collected data. Without this, understanding can be skewed. However, gathering metrics is only the beginning; the next step is to start asking “Why?”
- Articulating the value – Possessing data is one part of implementing software delivery metrics; but another challenge lies in using them to convey to the organization and teams the significance of their actions and why they matter. Failing to tie metric optimizations to business outcomes will lead to questioning the investment in resources needed to improve.
If these pitfalls sound familiar, perhaps there’s a gap in telling the story behind your data. This might be a signal for teams and organizations to prioritize crafting compelling and actionable stories from their software delivery performance data.
So how does VMware Tanzu Labs practice this approach to storytelling?
Storytelling of software delivery measurement at VMware Tanzu
At VMware Tanzu, measuring software delivery is a fundamental aspect of enhancing and innovating our products and services. We do this by completing the following steps:
- Selecting metrics as a baseline to support our storytelling method. We start with DORA metrics because of their prevalence in the industry and the research that supports their relationship to healthy outcomes.
- Capturing the raw data to build out the metrics.
- Converting the data into a consumable format that meets the organization’s needs.
- Telling the story and providing actionable recommendations that drive towards the organization's goals.
This article focuses on Step 4. The storytelling framework is intended to extract what matters to the operation of software development teams, and consists of three enabling factors: technology, process, and practices. These three enablers are created as a framework to help teams architect the stories that inspire the audience to make effective decisions and take appropriate actions. It is important to note that the framework is meant to be adopted and adapted to meet the specific software delivery goals of the organization. Hence, it’s meant to be flexible based on the context of use.
The three enabling factors of storytelling framework
Let’s take a deeper dive into the three enablers.
Software delivery metrics storytelling framework
Technology
Technologies are created to solve a problem or to influence a new behavior. VMware Tanzu offers various technological solutions to enable the software engineering ecosystem (e.g., VMware Tanzu Application Platform, VMware Tanzu Kubernetes Grid, and more). This part of the framework focuses on what the organization learned from adopting the current technology and identifying opportunities to scale or accelerate. Here are some questions that can help you tell this part of the story:
- What technological solution(s) influenced the results of the metrics?
- How does the technological solution(s) enable the desired outcomes or solve a problem?
- What technological solution(s) should be implemented to get the organization to their north star?
Process
Software development is an iterative process that creates products or services to meet business objectives. This part of the framework empowers storytellers to reflect on the strengths to drive towards business goals and the gaps that hinder the progress. Below are some sample questions that can be used to tailor the conversation about the process:
- What is the current process the organization has in place that influences the results of the metrics?
- How does the current process enable the desired outcomes or solve a problem?
- What part of the current process should the organization consider modifying to support an organization's north star?
Practices
Following software development practices leads to improved code quality, efficiency, and maintainability, thereby enhancing the ability to scale the operations in supporting the organization’s goals. VMware Tanzu Labs partners with clients in coaching team members and leaders with software engineering practices (e.g., extreme programming (XP), and more). This part of the framework zooms in on the ability to recognize the practices that enable the teams and organizations to be effective and hyperproductive on a daily basis. It gears the conversation towards the what and prevents organizations from unintentionally pointing fingers at teams or individual members on the team. Below are some prompts that can help produce healthy and actionable conversation about practices:
- What are the current software development practices that influence the results of the metrics?
- How do the current software development practices enable the desired outcomes or solve a problem?
- What software development practices should the organization consider adding or modifying to help them get to their north star?
Let’s dig into how the framework of the three enablers was applied to a real life situation.
Case study
VMware Tanzu partners with customers to solve problems that organizations are facing that can be solved with software. In this case, the VMware Tanzu team supported a federal organization with the goal to automate software delivery to their air-gapped remote edge server environment—a challenge familiar to many enterprises with edge servers.
The VMware Tanzu team, including platform and solution engineers, as well as solution architects, led the onsite team through the exercise with the goal to enable the client with the ability for the systems to exchange operational information in the right format, to the right person, at the right time, also known as interoperability.
To successfully enable the client with interoperability, the Tanzu team adopted the three-enabler framework.
- Technology – Tanzu Application Platform was adopted to manage the entire software delivery process, from solution development to deployment.
- Process – Continuous integration and delivery (CI/CD) tooling was utilized to support automation of the process.
- Practices – XP and SRE practices were adopted to aid the team in troubleshooting and updating new versions of the remote edge server site.
After implementing the three enablers to solve the interoperability challenges, the team began tracking the health of the software delivery process. DORA metrics were collected based on incident tracking reports. These metrics uncovered bottlenecks in the process and helped the team prioritize improvement efforts. Here is a sample of the DORA metrics captured within a month of the implementation of the three enablers:
- Deployment frequency (DF): 8 times per month
- Lead time for changes (LT): <2 hours over 1 month
- Mean time to recover (MT): 5 hours over 1 month
- Change failure rate (CFR): 0 incidents in 1 month
By using DORA metrics, the Tanzu team and client investigated the possibility of supporting the client’s business goal, which was to react faster with software changes at the edge. The metrics collected were used to help make software delivery improvement decisions and validate that the chosen technology, processes, and practices implemented were driving the desired results.
With measurement and a feedback loop in place, the team can now move forward developing each of the three enablers as follows:
- Technology– Leverage tooling built into the Tanzu Application Platform supply chain to automatically and iteratively capture DORA metrics.
- Process – The Tanzu Customer Success team can help facilitate metrics discovery workshops to identify the current state of the effort by highlighting the strengths and gaps in the current process; then create targeted actions that enable the software team and support business goals.
- Practices – Tanzu Labs can partner with the organization in order to coach the team with the XP skills needed to meet and exceed software delivery objectives.
Here at VMware Tanzu, we partner with some of the largest and most critical private and public organizations to enable their software delivery process and practices to meet and exceed organizational end states. Let us know how we can partner to accelerate software delivery transformation in your organization.
Get started
- Observe when your team or organization discusses the performance of software delivery, whether it’s a formal workshop or a coffee chat in the morning.
- Try our storytelling template using a three-enabler framework.
- Connect with Tanzu Labs to learn more about how the VMware Tanzu team can help!
