DevOps: Where Are We and How Did We Get Here?

Businesses today are well aware that the ability to drive differentiation through digital solutions is critical to their survival. However, decades of focus on cost-cutting and risk elimination have resulted in the perception of their internal IT organization as slow, overly burdened by policy and process, and incapable of supporting, much less driving, innovation. Technology is moving faster than ever, and IT teams are realizing that they need to adapt and grow or face becoming increasingly redundant as newer, faster technologies become increasingly available elsewhere.

One thing is clear – change is essential. The culture, operating models, and technology of today’s IT organizations all need to evolve to support the agility and speed required to deliver the applications and technology central to a business’s digital transformation.

This series of blog posts will examine DevOps as the cultural and operational shift for IT organizations to drive increases in efficiency and effectiveness.  It will consider where DevOps fits among the myriad of other technology initiatives chasing similar outcomes, such as cloud, cloud-native, Kubernetes, and other trends. (Spoiler: none of these initiatives stand alone, nor is it all or nothing.)

DevOps: How Did We Get Here?

DevOps is a term used to describe the collection of methodologies, frameworks, and practices that were created or evolved, in parallel and independently, with the goal of building better software more quickly and with the fervent cooperation between Development and Operational teams.

In recent years I rarely encounter an Enterprise, government agency, or educational institution that is not planning or already executing on a DevOps initiative. How and why did the movement gain such traction, and what does it mean for many IT organizations?

I find it helpful to review backward when assessing the way forward, so here is my quick, simplified history on how DevOps came to be.

The Pioneers

The term “DevOps” was first used by Patrick Debois after watching the seminal Flickr presentation at the O’Reilly Velocity conference in 2009. John Allspaw (SVP Operations) and Paul Hammond (Dir. Of Engineering) shared how increased cooperation between Dev and Ops enabled Flickr to deliver “10+ Deploys per Day”. This almost seems quaint compared to the hundreds of deploys per day currently achieved by Amazon, Google, and Netflix. At the time, however, the concepts they described were groundbreaking.

Patrick had already been working with Andrew Clay Shafer and others, brainstorming on how to make infrastructure more agile. He decided to organize a conference to collaborate on these emerging ideas and named it Devopsdays (which has grown from one event in 2009 to ~80 global events in 2019.)

Thus, DevOps was born. Although the name DevOps was new, much of what was being advocated was an evolution of existing practices, adding an intense focus on breaking down the barriers and silos that make team collaboration complex and ineffective.

First Came Agile

Traditional software development (often called “Waterfall” because the process cascades through stages, flowing mostly in one direction) involved the comprehensive definition and documentation of all application requirements before starting development. This point-in-time assessment/attempt to predict the future, coupled with the need to determine project scope and costs upfront, often resulted in inflexible timelines and financial plans built on guesswork. It is no wonder that IT projects became notorious for extensive delays, cost overruns, and – in the worst cases – a delivered solution not fit for purpose due to requirements that changed drastically along the way.

The writers of the Agile Manifesto (2001) realized that this was an ineffective way to build software and brainstormed on how to create a more efficient process. One that accepted that requirements almost always lack precision and seldom remain static over time. Their response was the definition of a set of software development principles that anticipate and respond to change by focusing on delivering functionality in smaller, more frequent releases with constant feedback between users and developers known as the 12 Principles of Agile.

This iterative, flexible approach to software delivery drew from existing and complementary methodologies that provided implementation options for Agile concepts. Many had themselves evolved from earlier methodologies, some with roots in manufacturing from decades earlier. For example;

• Scrum (1995) evolved from the Scrum method for product development (1986) and focuses on team collaboration for project success through the Three Pillars of Scrum – Transparency, Inspection, and Adaptation.
• Lean (2003) is the application of Lean manufacturing principles (which originated at Toyota in the 1940s) to software development. The 7 Lean Development Principles range from built-in quality and waste elimination to fast delivery.
• Extreme Programming (XP) evolved out of software development practices implemented at Chrysler in the 1990s aiming to reduce the cost of changes in requirements by having multiple short development cycles, rather than a long one.

In fact, representatives for these methodologies were the writers of the Agile Manifesto and Principles, creating an allegiance that set the foundation for evolving existing methodologies and creating new ones.

The adoption of Agile methodologies for software development quickly gained popularity as developers found that it resulted in improved product quality and increased productivity. However, this placed new demands on infrastructure and operational teams who were, after decades of ITIL and risk aversion, poorly positioned to adapt to these faster cycles and constant change. This resulted in increased frustration between the development and operational teams.

Enter “DevOps”

From the seeds of the success of Agile principles for software development, and out of frustration with the operational status quo, evolved the realization that these concepts should be extended to bring similar benefits to other parts of the IT organization, namely operations.

DevOps extends Agile principles to the Operational phases of the application lifecycle. The term DevOps itself is pretty self-explanatory – development and operations collaborating to make the creation, deployment, and maintenance of applications a more seamless, iterative, and cooperative process.

DevOps reached new levels of visibility in 2013, with the publication of “The Phoenix Project”. This novel by Gene Kim was able to capture the growing frustration between development and operational teams in a way that resonated with IT professionals everywhere. The release of “The DevOps Handbook” three years later was a collaboration between Gene and other DevOps luminaries and provided guidance and lessons learned from observing the outcomes of DevOps initiatives at high performing organizations. Suddenly, DevOps was on the lips of every IT executive and business leader as they heard about this new approach to managing technology.

The DevOps Loop

The most recognizable image associated with DevOps is that of the DevOps loop. The loop reflects visually the continuous flow between the development phases (plan, code, build, test) and operation (release, deploy, operate, and monitor) of an application lifecycle. This continuous loop is at the heart of the DevOps movement, and I will spend additional time examining it in the upcoming post “DevOps Culture – Collaboration, Shared Ownership, Empowerment”.

The DevOps Toolchain

Commonly, alongside “The DevOps Loop” are additional graphics that represent “The DevOps Toolchain” where various tools are mapped to the DevOps lifecycle phases (with the number of available tools increasing exponentially over time). Consequently, it can look like a hectic, overwhelming space where “getting started” can seem intimidating. The number of associated tools and frameworks is likely to continue to increase rapidly making the space even more crowded and confusing. I take a closer look at the tools space in the upcoming post “DevOps Technology – The DevOps Toolchain”.

DevOps at VMware

VMware lives DevOps in many ways; as a practitioner of the principles for software development, as a provider of tools and solutions that support DevOps practices, and as an advisor and implementor for DevOps initiatives across many of our customer organizations;

• VMware transformed to an agile foundation over a 3-year period, embracing a DevOps culture across our engineering teams and completing our DevOps transition in 2017.
• VMware solutions, such as vRealize and Tanzu, are an important part of the DevOps Toolchain ecosystem.
• VMware provides consulting and Professional Services to customers looking for assistance at any stage of their transformation journey.

Conclusion

My perspective is that DevOps is simply an umbrella term for a set of evolving practices with the objective of improving technology outcomes for businesses through agility, continuous feedback and learning, and with deep cooperation between development and operations resources.

IT organizations must change their approach to building and maintaining applications in order to compete effectively in today’s digital age and DevOps offers the latest and greatest recommendations for doing so. Many organizations have already started DevOps initiatives for select application teams, and some have adopted wholesale for application and infrastructure projects alike.

My upcoming blog posts will attempt to unravel some of the hype, clear up buzzword confusion, and provide a pragmatic, strategic approach to embracing DevOps for your IT organization.

Recommended Reading:

The DevOps Handbook (2016), Gene Kim, Jez Humble, Patrick Dubois, John Willis

The New Kingmakers (2013), Stephen O’Grady

War and Peace (2019), Mark Schwartz

Other Posts in this Series: