In an earlier post “DevOps: Innovators and Outcomes – The Disrupters” we looked at some of the organizations that drove the ideas and concepts that came to be known as DevOps, giving them a critical edge as they moved to disrupt existing markets and create new ones.
In this post, we look at organizations that were early adopters of DevOps; partly as a response to the digital disruption threatening their markets, adopting practices proven by the disrupters to enable agility and flexibility.
DevOps Early Adopters – The Disruptees
In 2011 Marc Andreessen predicted that “in the future every company will become a software company” as “software is eating the world”. This prediction soon came to fruition for the retail and banking segments where online shopping and banking alternatives threatened their traditional business models. For organizations in these markets technology was not their core business, but their use of technology would absolutely determine their continued viability.
The Disruptees found that they had to evolve or die, and digital transformation – gaining a competitive edge through software – was the only way to compete. In this post, we look at three organizations – Nordstrom, Target, and Capital One – who were pioneers in using DevOps principles to drive their digital transformations.
The Catalyst
Around 2011 Nordstrom experienced a trifecta of events that acted as catalysts for change:
- A major outage* – a highly visible, major outage to Nordstrom.com during an expected high-volume period exposing the lack of performance testing in the code release process.
- A failed in-store application rewrite – On delivery of a multi-year project to rewrite an in-store application (Personal Book) it quickly became apparent that it was already irrelevant, failing to connect with customers and sending the teams back to the drawing board.
- A Strategic Initiative – In 2011, the Board of Directors at Nordstrom decided to focus on online growth as a strategic initiative, applying pressure that would exacerbate existing problems in development cycles if not addressed.
*The logical response to the lack of load testing was to add performance testing, which did help to catch potential load issues in subsequent releases. However, it also required a significant increase in capacity, added additional time to the release process, and did not scale across operational teams.
Enter DevOps
The operations team realized that the performance testing environments were a temporary and expensive fix, testing application code retroactively just prior to release. The ideal solution is for quality and testing to be embedded throughout the development process. A small team was assembled to implement recommendations coming out of the burgeoning DevOps movement, transitioning to infrastructure as code and embracing continuous delivery. Their journey and lessons learned are documented in the O’Reilly case study DevOps in Practice.
The application team – as a result of the failure of the “Personal Book” rewrite – realized that they needed to move to an incremental development process; one with an ongoing feedback loop to regularly validate features and requirements, and flexible enough to allow for changes as needed. Application teams across Nordstrom were directed to use the Agile methodology to speed up the development process. Some teams quickly embraced and benefited from Agile practices, while others simply paid it lip-service. Consistent success came only when leadership embraced an Agile principle that had been missed initially – let the teams drive the change that affects them, rather than dictating it top-down.
Central to the success at Nordstrom became the concept of “value-stream mapping”. Teams would take value outcomes (such as an update to an iPhone application) and attempt to model the flow of work required to deliver. These mappings identified handoffs and delays, and illustrated the delta between “work as perceived” and “work as performed”. Visualizing the value-stream allowed teams to better understand the entire flow of work and collaborate to optimize processes holistically instead of locally.
Outcomes
Like many organizations, the 1,800-person IT organization at Nordstrom was seen as a cost center and optimized accordingly – for cost. Embracing DevOps allowed the technology organization to transition to operating as a business partner, optimizing for speed. Outcomes included:
- More frequent feature releases (~28 weeks to 4 weeks or less)
- Higher quality updates (faster deployments, fewer bugs and rollbacks)
- Reduced “fire-fighting” and increased planned work
- Increase in employee satisfaction
- Doing more with less – supporting more complex, larger environments with the same resources
So successful were the DevOps and Agile processes adopted at Nordstrom that they were able to apply them to other areas of the business. At the DevOps Enterprise Summit in 2014, they shared how value-stream mapping can be applied to any team delivering value, not just those that needed to move quickly, resulting in the elimination of waste, streamlined cycles, and a culture of continuous improvement.
Recommendations
- Start small and expand organically
- Let the teams doing the work drive the change, with leadership providing air-cover with support and funding
- Embed “shared services” roles into the DevOps teams, with application developers and QA resources
- Expect some ideas to fail – provide a “safe space” for trial and error
- Create “value streams” to map the flow of work through the system
- Transition to outcome-based measurements
The Catalyst
In 2013, the now-infamous data breach at Target exposed its technology organization to criticism and intense scrutiny. The resulting evaluation revealed challenges that are common across many IT organizations:
- Lack of a dynamic engineering culture (70% outsourced)
- Complex organizational structure (siloed engineering teams far removed from customers)
- Complex IT systems (massive technical debt)
Target prioritized rebuilding its engineering culture; to champion excellence and value talent, attempting to evolve into a world-class, high-performing technology organization.
Enter DevOps
The DevOps journey at Target started with a small, grassroots team, supported by a couple of key change agents in leadership. Inspired by the Phoenix Project, the team in 2011 was focused on “automating everything” and implementing continuous integration for software development.
By 2014, when Target shifted focus to rebuilding its engineering culture, this team had already proven the value to be gained from DevOps, and were important contributors to the wider DevOps movement. They had recently held the first internal DevOpsDays, aiming to build a community connecting the expanding grassroots efforts across internal silos to share and learn from each other.
A new CIO in 2015 brought additional leadership focus to improving cooperation between development and operational resources and striving for a more agile software development process. In order to scale from the grassroots initiatives into an engineering organization transformed by DevOps principles, Target invested heavily in training and workshops. In 2015, Target created “The Dojo”. The Target Dojo coaches teams through 30-day challenges where they transition from traditional delivery to using DevOps and Agile principles. Teams enter the Dojo with processing times of three-to-six months to provision environments, and they leave capable of doing it in a few hours.
To provide flexibility in service delivery, Target created teams built around infrastructure components, with each team providing API access for other teams to consume components and stitch together environments. The teams have full-stack ownership for the API toolset, with operational resources playing a key part in the development process.
Outcomes
Target was successful in rebuilding its engineering culture, attracting engineers, and increasing innovation. They successfully shortened release cycles, improved application quality, and increased employee morale. Target considers its DevOps movement to intersect between the three values required for the seamless delivery of product:
- Collaboration
- Empathy
- Experiential Learning
There are now three Dojos (the immersive learning environment) across the world, 37 coaches, and 265 completed challenges (up from 14 in 2014). DevOps has transformed the entire culture at Target.
Recommendations
- Align incentives across development and operational organizations
- Change the way you see and treat failure – fear of failure will stifle innovation
- Prioritize continuous learning – experimenting, testing, failing, and ultimately succeeding
- Shift to a “blameless culture”
- Focus on results and capture metrics, e.g. faster delivery, better quality, overall lower costs
- Build trust with peers across the organization through communication and transparency
The Catalyst
While not necessarily a catalyst tied to a specific event, Capital One considers itself a start-up in the banking community (being a relatively youthful 20 years old). There is an entrepreneurial spirit that is embraced and encouraged. Inconsistent releases, manual steps, multiple handoffs, and environments that took months to build did not support that spirit. Capital One knew that they wanted to deliver high-quality, working software faster, and when new approaches started to emerge in the form of DevOps, they jumped on it.
Enter DevOps
DevOps at Capital One started with a small team and, once they saw processes that went from days to minutes, the need for an enterprise-wide strategy became clear. Capital One approached their initial DevOps initiatives in phases:
- Phase One: Add configuration management and automated middleware and application delivery to application environments
- Phase Two: Move infrastructure to the public cloud, implement full infrastructure automation with infrastructure as code, and automate delivery pipelines
- Phase Three: Implement robust delivery pipelines, including automated quality tests, and pre-approved releases
The transition was kicked off by assigning dedicated, cross-functional, “SWAT” teams to two legacy applications. These teams worked consistently for two months at a time to implement configuration management, automate critical processes, and improve the flow of work for a given application. At the end of the engagement ownership of the software delivery process was transitioned back to the application teams.
Capital One completed the transition of four additional applications this way, before allowing the application teams to drive and implement the changes themselves using the best practices identified by the SWAT teams.
As of 2017, Capital One had followed the lead of Netflix and implemented their own version of “chaos engineering” to select applications, including a disruption tool called “cloud detour” to test application resiliency against a variety of failures and outages.
Outcomes
As early as 2015, Capital One was publicly announcing that the adoption of a DevOps culture had significantly improved the software deployment process for multiple applications, including:
- 100s of code commits per day,
- Integration from once a month to every 15 minutes
- QA from once per month to 4 times per day
- Deployment from manual to completely automated
- Production release from monthly/quarterly to once per sprint
Today, Capital One is a major contributor to the DevOps community, the Open Source community, and regularly shares insights from its software engineering practice publicly.
Recommendations
- The initial use of cross-functional SWAT team helped to create shared goals, remove handoffs, and learn developer/operational disciplines
- Moving the completed automation back to the application teams exposed the need for additional training for application teams
- Automation is critical, but don’t try to automate everything at once
- Implement a virtual development “clean room” with clearly defined guidelines to safeguard the deployment process
- “Don’t let great get in the way of being very good”, if you wait until things are perfect you will never deliver anything. Deliver something, get feedback, improve it
Conclusion
These early adopters embraced DevOps in response to the threat of disruption in their markets, in order to support the digital transformation initiatives that would allow them to remain viable. While there were some existing pockets of grassroots DevOps initiatives, the prioritization of technology transformation on a larger scale did not occur without a catalyst – some kind of highly visible, disruptive event. Too often in technology, organizational change comes as a reactive response to an outsized disaster. Could these issues have been avoided by adopting DevOps sooner?
Many early adopters embraced the DevOps community by contributing to the movement and sharing experiences and best practices. As we move into mainstream adoption of DevOps, there is a wealth of information and data available to help organizations succeed quickly – experiences shared by those that have already adopted and benefitted from the transformational approaches that DevOps brings to technology teams.
There are some recommendations that we hear consistently from organizations that have transitioned. These include the role of leadership (to provide guidance and support), the need to embrace failure as a critical part of the innovation cycle, and the importance of making teams responsible for their outcomes through empowerment and autonomy. I discuss these in more detail in my next post, DevOps: Culture – Collaboration, Empowerment, Autonomy
In the meantime, don’t wait for the next technical disaster to start your transformation! Reach out to your VMware team to find out how we can help.
Recommended Reading:
Accelerate: The Science of Lean Software and DevOps (2018) Nicole Forsgren PhD, Jez Humble, Gene Kim
Other Posts in this Series:
| DevOps #1: Where Are We and How Did We Get Here? | May, 2020 |
| DevOps #2: Innovators and Outcomes – The Disrupters | May, 2020 |
| DevOps #3: Early Adopters and Outcomes – The Disruptees | June, 2020 |
| DevOps #4: Culture – Collaboration, Empowerment, Autonomy | June, 2020 |
| DevOps #5: Devopsdays – DevOps Culture Embodied | July, 2020 |
| DevOps #6: Principles and Outcomes | August, 2020 |
| DevOps #7: Technology – The DevOps Toolchain | August, 2020 |
| DevOps #8: Continuous Everything | September, 2020 |
DevOps at VMware
VMware lives DevOps in many ways; as a practitioner of the principles for software development, as a provider of tools and solutions that support DevOps practices, and as an advisor and implementor for DevOps initiatives across many of our customer organizations;
- VMware transformed to an agile foundation over a 3-year period, embracing a DevOps culture across our engineering teams and sharing our journey with customers.
- VMware solutions, such as vRealize and Tanzu, are an important part of the DevOps Toolchain ecosystem.
- VMware provides consulting and Professional Services to customers looking for assistance at any stage of their transformation journey.
