agile culture devops DevOps Best Practices Enterprise Strategy management Platform Engineering Best Practices Thought Leadership

DevOps Loop Recap: A Day Filled with Glorious Purpose

DevOps Loop at VMworld, a new community event launched this year, set out to examine DevOps and its core principles in the context of modern apps, multi-cloud, and Kubernetes. We pulled in all the experts—those folks on the ground living and breathing DevOps—to share their viewpoints. And the day was AWESOME. All the sessions are available to view (or view again) on demand. I’m always surprised by how much I miss on the first pass. 

Here are a few DevOps insights I gleaned from the day:

  • Change is required – Modern apps and multi-cloud can make it easier and harder to deploy to production. How can you rethink people and processes to meet the change?

  • Cognitive load is real – Complex technologies and architectures are pushing folks to their limits and actually impeding progress (hello, CNCF landscape!). How can you lighten the load?

  • Confidence is lacking – How can you gain trust in your processes and people to deliver software fast and frequently—and reach the pinnacle of DevOps success?

Let’s explore some answers.

Evolve DevOps, and make change happen

In many ways, it’s easier to build and release a modern application. You’re building small, loosely coupled applications and services that can be released independently (and daily). Contrast this with monolithic applications that must be updated, tested, and released as a whole entity (and you might update quarterly or even less frequently because it’s so hard). 

Releasing a small thing is so much easier than releasing a big thing. But it also comes with new patterns and technologies and tools and roles that are needed to ensure these distributed apps are secure, scalable, and meeting service level objectives. For organizations to adapt to the new cloud native world, many of the speakers at DevOps Loop suggest we need an evolution (if not a revolution). 

Joe Beda, one of the Kubernetes founders, kicked off DevOps Loop talking about evolving DevOps and the fundamental shift that must happen—moving from providing infrastructure services to providing a product for developers. Get rid of ticketing systems and humans in the loop. Replace them with a self-service, API-driven platform that can accelerate application teams.

This necessitates rethinking roles as well. If you have a platform as a product, you need operators and developers for that platform, with developers focused on feature delivery. For your applications, you need not only traditional app developers, but you also need an app operator—someone fundamentally driven to ensure the application is running well on the platform. And the platform and application teams must work collaboratively to make sure the platform delivers true business value. 

The one slide you should capture from Joe Beda’s presentation

Then Emily Freeman, DevOps solutions principal at AWS, presented a complete reimagining of the software development lifecycle. She posits that “DevOps is an increasingly outmoded solution to a previous problem. Developers now face cultural and technical challenges far greater than how to more quickly deploy a monolithic application. Cloud native is the future, the next collection of default development decisions—and one the DevOps story can’t absorb in its current form.”

The Revolutionary Software Development model from Emily Freeman

Freeman proposes a new model that builds on the foundation of DevOps and agile. She calls it the Revolutionary Software Development model, designed to more accurately reflect the complexity of modern software development. I think the exciting thing about this model is that it, in many ways, de-emphasizes the hard divides between different roles and instead focuses on core capabilities that all roles should take on from their different vantage points. You’ll be able to see this model in action in her session.

In his lightning talk, Getting Security in the Loop: Building Balanced Teams, David Zendzian, global field CISO for VMware Tanzu, talked about how DevSecOps is more about helping security evolve. Security teams are still doing the same type of threat modeling—a lot of paperwork and processes for documenting risk and making decisions. Their processes are cumbersome and time-consuming. Zendzian suggests bringing security as a function into the application lifecycle where they can learn the language of the development team, help secure applications, and aid in automating security into the lifecycle.

Security and development teams must work together to lighten the cognitive load

If an organization is not changing how it works to embrace modern apps and cloud technology, is it doomed? In his session, We Fear Change, Michael Coté, developer advocate at VMware, said that many executives take a “change or die” approach to transformation—there’s no question in their minds that changing how their organizations work is critical. But, Coté says, change resisters are not going to change just because you’ve shown them the truth. He goes on to provide some strategies for motivating change, like fulfillment (what they can professionally accrue from the change), safety (providing a supportive environment that facilitates change), and lifestyle (personal dividends derived from the change).

Lighten the cognitive load using DevOps core principles

While change is in the air, the vastness of the technology landscape seems to be weighing development and operations teams down. As Bola Rotibi, research director of software development at CCS Insight, told us in her session on DevOps in the Real World, organizations need to embrace the future while continuing to do business in the here and now. It’s not an either or choice; no IT environment is homogenous. 

Many speakers at the conference talked about how the act of applying DevOps principles—culture, automation, lean, measuring, and sharing—can lighten the load. Dr. Nicole Forsgren, partner at Microsoft Research, covered this in her session, Why Even DevOp? She says that the savings you get from DevOps “gives us brain space to think about our next problem or our next project. It creates space for us to do something fun.”

Forsgren cites data from the 2021 Accelerate State of DevOps Report indicating that a  positive team culture can mitigate burnout during challenging times. “Making these investments really help us improve our own well-being. Improving process, culture, and automation is key.”

A positive team culture, along with process improvements and automation, mitigate burnout

Forsgren adds, “Automation is not a threat. Automation is an opportunity. It frees us from this mind-numbing work and saves us from the cognitive load that we have.” Her session also detailed what productivity looks like (and guess what, it’s personal!). 

You have to be wary of the cognitive load you’re putting on developers when it comes to Kubernetes. What is the right level of abstraction to eliminate unneeded overhead? In the panel, Kubernetes: Who’s Idea Was This?, we heard a lot of viewpoints:

  • Bryan Liles, a principal engineer at VMware, said that in an ideal world your apps shouldn’t have to know that they are running on Kubernetes directly. 

  • Joe Beda told us “it depends”—on who the developers are and what the applications require. He also commented that there is no single, simplified abstraction that’s going to work for everyone, and no abstraction that will work for everyone over time. 

  • Andy Burgin, lead platform engineer at Sky Betting and Gaming, certainly wants to abstract storage and networking. But he recognized some developers want direct access to Kubernetes and others just want to write four lines of YAML to deploy their code. 

  • Josh Rosso, staff architect at VMware called this the “cat and mouse abstraction game,” a hard problem to figure out.

Security also adds cognitive load to organizations. Common vulnerabilities and exposures grow every year, and modern, distributed applications can increase the attack surface. You’ll hear about shifting security left, but does that mean developers must now become security experts too? In the panel, It’s DevOps (the Sec is Silent), the panelists talked about the security divide, and how more sharing and collaboration needs to take place. 

As Dominique Top, solutions engineer at HashiCorp, stated, “If you’re doing security training once a year, you’re not going to change anything. You need regular meetings. If you want to influence change, you have to influence behavior.” She continued that security needs to provide examples of how things can go wrong, and the implications for not doing the right thing. David Zendzian added that every security team he’s worked on has wanted to be more involved in application development as early as possible to help reduce risk and protect the data—to help build the right hooks into the application for security teams. After all, he said, “no developer wants to have unsecured code that’s published.”

The bottom line is that no one should have to go it alone. In an ideal setup, security is partnering with development and operations to figure out how to make the right thing the easy thing—and in the process, reducing the cognitive load for everyone.

And one final note. As a former technical writer, I am a firm believer that great documentation can help reduce cognitive load. I thought Heidi Waterhouse, transformation advocate at LaunchDarkly, hit the mark in her lightning talk, DevTechDocsOps, saying, “documentation is just guardrails for all the things that go wrong in life.” She points to the Accelerate State of DevOps 2021 research as well, highlighting that internal documentation is shown to help teams go faster. 

She covered some useful types of documentation:

  • Pre-automation planning documentation

  • Socio-technical processes (i.e., people processes)

  • Docs you read by flashlight (i.e., what you must know to get your system back up)

By the way, Waterhouse has a newly published book, Docs for Developers, that you should check out.

Build confidence through doing DevOps

To truly embrace modern apps, cloud native tech, and Kubernetes, you must embrace the idea that your deployments are going to fail and your apps are going to go down. But you’ve built them to be resilient, and your systems can help alert you and automatically get them running again. But this takes some level of confidence in your DevOps systems and practices. DevOps Loop speakers addressed the issue of confidence—of why we are not more bold and trusting—and how to become more confident.

Kat Cosgrove, staff developer advocate at Pulumi, in her session The History of CI/CD, took us back in time to learn about continuous integration (CI) and the two flavors of CD (continuous delivery and continuous deployment). She noted that continuous delivery is not enough because there’s still a human quality gate before code is pushed to production. 

Cosgrove is a big fan of continuous deployment, where the deployment to production is automated with no human intervention, which requires a lot of trust in your system. She says that, “for your DevOps pipeline, and thus you, to be as efficient as possible, we have to remove human involvement wherever we can.” Her advice: write good, comprehensive tests, automate everything you can, and then accept that you are going to deploy a bad update eventually. What matters is how quickly you can respond to a bad update. 

The discussion of why folks are not doing continuous deployment continued into the panel discussion Making CD Work in the Real World. Bola Rotibi acknowledges that it’s a confidence thing—that the consequences for getting something wrong are high. Kat Cosgrove adds that it’s the confidence in the automation that’s lacking. 

The panelists suggested many ways to build confidence in your CD process:

  • Kat Cosgrove says to start small, automating more and more things to build confidence over time in the quality of your automation. Iterate on the automation and get more comfortable with it.

  • Jason Yee, director of advocacy at Gremlin, mentions that you must discover where you lack confidence. For example, if you are afraid of rolling out software that will break in production, then you can build up the confidence in your monitoring. You should know immediately if you deploy bad code. And if you detect problems quickly, you should be able to roll it back. 

  • Cornelia Davis, product management at Amazon, adds that “you can’t change minds on faith.” It’s valuable to demonstrate failure and the safety nets that are saving you. Show your teams what happens when things go wrong. Show the metrics of fixing the problem—like mean time to recover. She believes that safety nets introduce a level of comfort. 

  • And Bola Rotibi reminds us that we’ve been doing this job for a while, and to “have a little belief in yourself and your team.” You may lag in confidence because the tools and capabilities are not there yet. So invest in the right tools, and start off small.

In her session, The Business Benefits of GitOps, Cornelia Davis showed how to gain confidence in your infrastructure through a GitOps approach. Davis does an excellent job describing GitOps, including how to:

  • Take advantage of Git review capabilities for configuration code

  • Deploy new configurations progressively

  • Detect and remediate drift problems

  • Restore a previous version with confidence

It’s the beauty of convergent systems, like Kubernetes, that can bring extensive value for continuous operations.

With modern applications, and their many moving parts, maybe you lack confidence because you can’t see around corners. In that case, watch Jason Yee’s lightning talk, The Adjacent Possible. Yee says, “as applications become more distributed and more complex, I believe that there is an adjacent possible of failure.” To solve for this, he says that we must be proactive about failure and reliability. Part of this is understanding what you know and what you don’t know about potential failure in your system (and how to mitigate it). 

Chaos engineering helps expose your unknown unknowns

But you don’t just have to rely on hope! Yee is a big advocate for chaos engineering because it helps organizations explore those unknown unknowns and turn them into known knowns. Then you can learn how to remediate problems faster and improve your systems. 

Learning never ends: Tanzu Community Edition is your K8s playground

One of the best ways to gain confidence is by learning more about cloud native technologies and Kubernetes—and all the possibilities they bring. That’s why VMware was pleased to announce the release of VMware Tanzu Community Edition at DevOps Loop. 

During a quick session, Amanda Katona, director of engineering at VMware, and Josh Rosso, technical lead for Tanzu Community Edition, shared a first glimpse of this new, free Kubernetes distro. Katona talked about how this community-supported, open source distribution can be installed and configured in minutes on a local workstation or favorite cloud. She underscored its suitability for users and learners alike, and shared a quick taste of what’s available through the community, including education and support resources.  

Rosso demonstrated a few key features of Tanzu Community Edition. He showed how Tanzu enables you to manage clusters in the same, efficient way you manage Kubernetes workloads, using declarative automation. He also demonstrated simple and elegant package management capabilities that can be used to easily customize your Kubernetes platform. 

Get access to all the DevOps Loop sessions and panels here:

Related Articles