The companies we at VMware Tanzu work with are constantly looking for better, faster ways of developing and releasing quality software. But digital transformation means fundamentally changing the way you do business, a process that can be derailed by any number of obstacles. In his recent video series, my colleague Michael Coté identifies 14 reasons why it’s hard to change development practices in large organizations. In this companion series of blog articles, we’re exploring each of those topics in more depth, providing advice on how you can address them in your business.
Today, let’s look at the fifth digital transformation challenge: CI/CD bottlenecks.
Where does the time go?
A large part of my role as an executive advisor is to help large organizations reduce the time it takes to get new applications or features into production. I’ll commonly arrive to be told that they’ve already undergone a digital transformation, brought in lots of great automation, yet it can still take weeks or even months for a developer to move from a new requirement to code running in production. Where does the time go?
Usually, the slowdown comes from a neglected path to production. By “path to production,” I mean all of the activities that it takes to go from a new idea or modification to your app to a person using the app in production. The most basic path to production is specify -> code -> test -> release. Typically, in enterprise applications, there are many—many—more activities.
The critical flaw I repeatedly see in a path to production is that each team is optimizing their own part of the path, without enough understanding of how they fit into the entire, end-to-end process. There’s rarely enough time spent optimizing the entire process as a whole; most of the effort is spent optimizing each team’s “box,” rather than how those boxes all flow together. This is a classic problem of local optimization holding back overall optimization. Put another way, automation is great, but are we solving the right problems, and in the right places?
Charting the path to production
The purpose of a path-to-production workshop is to identify all the steps and teams involved in turning an idea into something tangible that delivers value to the business. A successful workshop needs representatives from teams across the organization—finance, PMO, security, architecture, developers, operations—anyone who plays a role in the software delivery process. Armed with a box of sticky notes and an empty wall, the group builds a timeline of every step in the process, from identifying a new requirement through having that feature implemented and in operational support. Next, the group identifies the teams responsible for each of those steps. Finally, the group can estimate how much work is associated with each task, and the average wait time as a task moves from one team to the next.
We can calculate the efficiency of the overall process by dividing the “working time” by total “elapsed time” (i.e., working + waiting). This gives us a metric we can measure going forward to chart real-world improvements in software delivery velocity.
A workshop like this is a great way to bring teams together and help them build a picture of the obstacle course of processes, team handoffs, and approval steps each new feature must navigate as it progresses from an idea to a funded project and from code on a developer’s laptop to various testing environments, until it eventually arrives in production.
Approval steps are often an eye-opening experience for the group because a few minutes of review can easily add days or weeks to the overall delivery timeline. Likewise, tech teams might find that their automation efforts would be best served creating clean development environments rather than trimming a few minutes from production deployment scripts. The value of an automated CI/CD pipeline (CI/CD stands for continuous integration and continuous delivery or continuous deployment, depending on whom you talk to) is often in removing handoffs between teams, rather than acceleration of a particular process step.
Once everything is up on the wall, you’ll often hear people say, “I’ve never seen this whole process before.” That’s usually followed by them muttering a sad “wow.”
But this is a good step. You can’t optimize what you can’t see, and now, you can finally see it all.
Four ways to accelerate your software delivery
Keep focus on the big picture. Developers must remember that the pipeline involves much more than just deploying an application; likewise, platform engineers must see past production and understand that most developer challenges lie in creating good testing environments that enable them to iterate quickly.
Find the toil. During the path-to-production analysis, it’s often useful to ask everyone to mark three areas that generate the most pain for them. Focusing on these gives the group some early wins where it matters, motivating them to tackle perhaps more challenging areas of process improvement.
Focus on efficiency. Businesses will find the largest gains in velocity by focusing on efficiency and eliminating wait time wherever possible. Tackling changes to process or challenging the need for antiquated approval steps can be difficult, but this a critical step in making developers more impactful and demonstrating that the organization is serious about reducing toil, improving developer productivity, and lowering time-to-market to deliver a differentiating edge for business revenues.
Choreograph your orchestration. Building CI/CD pipelines in each team is wasteful and leads to large gaps in maturity across the organization. On the other hand, a centralized approach often fails because it’s too inflexible—falsely assuming that all teams can work the same way, and that the needs of the teams won’t change too much or too quickly. Using a tool like VMware Tanzu Application Platform brings teams together by providing a sensible framework, but gives teams the flexibility to include their chosen automation tools and create a “golden path” that makes sense for their application.
Before anything else, find and fix the path to production
If you’re one of those organizations that is looking for new, better ways of developing and releasing software, start by looking at your path to production. Chances are high—almost certain!—that it can be improved. No matter what new tools or cloud platforms you put in place, that path to production will still exist, so it’s best to start looking there. You’ll likely find that you had no idea how complex and tedious it is to get software out the door. On the bright side, by conducting a path-to-production workshop, you’ll also likely find some quick ways to start improving how you do software. Before you buy a Ferrari, make sure you’ve got excellent roads.
Learn more
Here are some sources for additional reading on analyzing and designing an effective, secure path to production:
- Why Path-to-Production Analysis Is a Must for Software Development Teams
- White Paper: Developer Toil, the Hidden Tech Debt
- What Is Supply Chain Choreography, and Why Should You Care?
Read about other common digital transformation bottlenecks we’ve covered in this blog series:
- Securing Your Environment with Tools Before Rules
- Making Compliance a Feature, Rather than Friction
- 5 Ways to Cure Your Tech Debt Crisis
- Platforms as a Superpower
- To Succeed at Digital Transformation, Do Less
- Bottom-up Change Doesn’t Work
- How to Solve the Skills Problem
- Transformation without Belief Is a Waste
- Empowerment without Clarity Is Chaos
- Crusties Who Are Reluctant to Change
- Lack of Trust: All Talk and No Tools
- You Can't Change the Organization
Need help? Tanzu Labs has partnered with countless organizations across the globe to deliver applications and features to production faster. Tanzu Labs offers bespoke consulting services to help organizations build new or modernize existing applications, ensuring that best practices are transferred to customer teams along the way. Learn more about Tanzu Labs’ application services.
