Security is hard to get right in a world of continuous delivery and containers. The increasingly diverse technology landscape and relentless speed of innovation afford us no time to step back and take stock of our risks, and even less time to perform remediation. In the past, we could perform point-in-time security audits, which was OK when our systems were mostly static, save a quarterly release. But that’s not the world we live in today. According to the National Institute of Standards and Technology, for example, there are now more than 149,000 CVE vulnerabilities, a 20 percent increase since 2019.
We used to build walls between Dev and Ops; the problems created by those walls are what gave birth to the DevOps movement. However, we still often see security separated from the teams delivering software. To ensure we maintain robust security without compromising development velocity, we need new approaches, engineering practices, and tools. But most of all, we need to encourage better collaboration between security specialists and delivery teams.
The challenge: security vs. velocity
Unlike investments in resiliency and scalability, where the benefits are felt by the engineers on-call, the benefits of robust security practices are often invisible.
The absence of security incidents is not evidence of good security practices.
One of my colleagues used to compare patching to flossing. We all know we should do it because it’s good for our long-term oral health. But many of us (myself included!) often choose to skip it, because it’s a chore. There’s no immediate reward; you do it to prevent bad things in the future. A lot of security improvement work falls under this category of chores, and so can get deprioritized. It’s human nature to avoid chores, especially if there’s more interesting, rewarding, or urgent work to do.
Security and velocity are opposing forces in most of the enterprises I work with at VMware Tanzu Labs. Security processes can create friction and slow down development as well as make operations tasks more difficult. This breeds a culture where backdoors, workarounds, and shortcuts are common, and where talking openly with security specialists is discouraged.
If this sounds like your team, then there are two key elements to consider when you’re looking to improve your culture around security and make meaningful, sustainable improvements:
-
Ensuring psychological safety for any team members who want to talk about and/or report security issues without repercussions.
-
Taking a user-centered approach to the implementation of security policies and processes.
Changing the conversation about security
Talking about security is going to involve a calculation about risk and reward for each individual. If a team member reports a security issue, they need to be praised for their diligence—not scrutinized for their mistakes. Creating psychological safety for teams to talk openly about errors, mistakes, and known issues will encourage more people to do so.
Security doesn’t need to be a taboo subject.
Like a blameless post-mortem after an incident, we need a blameless culture around security to ensure we learn from mistakes and continuously improve. To create a blameless culture around security, be sure to:
-
Educate and listen – Are your security specialists available and willing to answer questions about best practices? Having empathy with beginners and creating space for anyone to ask dumb questions is really important for learning. Are security specialists proactively engaging the rest of the organization with easy-to-understand content and recommendations to help teams improve? Doing so can mean an operating model change for some security teams.
-
Measure progress – Agree where you want to make improvements and how you’re going to measure the results. What is the mean time to recovery (MTTR) from a security issue, for example? It’s also useful to start with a set of realistic and aspirational targets using tools like SLIs, SLOs, and error budgets. We coined the term vulnerability budget expressly for this practice.
-
Celebrate success – Now that you’re talking about problems, agreeing on solutions, and implementing those solutions, you need to make sure you do all of it visibly. There should be a sense of achievement for everyone! Some security issues may be quite sensitive while they’re still active, but there’s nothing stopping you from sharing the journey and lessons learned afterwards, and it will help prevent other teams from making the same mistakes.
-
Set an example – If you’re in engineering leadership or are a security specialist, you need to start by modeling the behaviors you want to see more of, such as championing honesty, supporting collaboration, and learning from mistakes. Be honest about the mistakes you’ve made and others will be more willing to speak up.
If you suspect your company or team is in denial about security, I highly recommend reading Our Iceberg Is Melting: Changing and Succeeding Under Any Conditions (Penguin Random House, 2016) by John Kotter and Holther Rathgeber. It’s a short read, and provides an excellent parable about how to approach change, especially how to lead change when the majority believe everything is fine. Poor security practices can often feel like a disaster waiting to happen—much like the melting iceberg described in the book.
Make it easy to do the right thing
Velocity and security are not mutually exclusive! Maintaining both is a hard nut to crack, but it can be done by focusing security specialists on the teams that are implementing their processes and adhering to their policies—in other words, their users. Approaching the design and implementation of security mitigations with a user-centric approach essentially means making it easy to do the right thing.
By making it easy to do the right thing, you’ll be secure by default.
An environment where everything is secure by default and helps teams get their job done will dramatically improve your security posture and encourage more collaboration between the security team and its users: developers and operators, as well as infrastructure and platform teams. If a security improvement negatively impacts productivity, then at best you’ll breed resentment, and at worst you’ll inadvertently encourage teams to find their own solutions, which may be less secure. And you might be impacting employee experience and contributing to attrition in the process.
I would argue that security initiatives implemented without a user-centric approach will degrade your security posture over time, as teams will figure out ways to work around the process or policy that slows them down. With that in mind, the following steps will help you and your team adopt a user-centric approach:
-
Automate toil – Making it easy to do the right thing often means automating the chores away. Bake security into your developer workflows, continuous delivery pipelines, environment provisioning, and access management processes so that you can minimize any additional effort or overhead associated with maintaining security and compliance.
-
Make ownership shared – Giving teams the autonomy to implement their own security practices also helps create a sense of shared ownership. You can do this by embedding a security specialist in your development or platform team for a period of time and having them work with the team to implement the most important improvements, which will enable them to build the relationships and trust they need to collaborate with each other in the future.
-
Understand developers – As continuous delivery and containers change the way we deliver apps, security specialists need to be aware of the impact it has on their users. Frequent releases might initially seem impossible to manage when a security process includes manual gates on the path to production. But spending time with development teams will help security specialists understand that repeatable, automated, small releases actually reduce risk, and patching rapidly and automatically will be an advantage when remediating security issues in the future.
-
Take a product mindset – Feedback loops are critical for your long-term success. Continuously solicit feedback from your users (remember that as a security specialist your users are development, operations, infrastructure, and platform teams; you probably have a lot of users!). By keeping the conversation open you can also learn about new technologies and trends that might prompt you to adapt your recommendations and adjust your approach. Measure your own cycle time from idea to production for policy changes!
-
Decide whether to build or buy – There’s an explosion of interest in DevSecOps and some really exciting tools and OSS projects available to help build a DevSecOps toolkit. It can be tough to know where to start, so I would recommend prioritizing based on your areas of highest risk, but also considering your users and how you can make their lives easier. Find a win-win on both security and productivity! If you do that, development teams will be more likely to want to collaborate in the future.
Wasn’t this part of DevOps all along?
I’ve been involved in the DevOps movement for a long time now; it’s been almost 10 years since I managed my first DevOps team (although we didn’t call it that at the time). In the early days, we talked about shifting left with security, but at the time we had limited tools and techniques available to make it a reality. Compliance-as-code has been around for a while—I remember the acquisition of Volcanosec and the launch of InSpec at the Chef Community Summit back in 2015, for example. But although InSpec was a fantastic tool, it needed to be implemented as a standalone solution, and required a significant investment of time to implement and maintain. For most teams, the effort vs. reward didn’t add up, which was a shame, as I worked with teams doing some amazing test-driven infrastructure as code using InSpec.
DevOps is about culture—so is DevSecOps.
Today, however, everything about software development is more complex, so we need to shift left. When implementing continuous delivery pipelines and a path to production for your containerized applications, for example, you need to be thinking about how secure your software supply chain is. What’s in your container? Where did it come from? How are you going to patch it quickly and automatically if someone finds a CVE tomorrow? How can you ensure that communication among containers and microservices is secure?
These are the real-world problems we’re solving at VMware through a number of products and open source projects designed with DevSecOps in mind, such as Harbor registry, Tanzu Build Service, Tanzu Application Catalog, Carbon Black Container, and more. Kubernetes is also used to provide scalable and resilient application platforms, but we need to ensure we’re automating the provisioning, policy management, and upgrades for our clusters. These are often seen as Day 2 concerns and so are deferred to a later date, but they’re actually hugely important and something we’ve made much more simple with Tanzu Mission Control.
However far technology has come, though, the fact remains: A tool can’t fix your broken culture.
As your team starts to explore this space, remember to consider the human aspects of a DevSecOps transformation. Be sure to create the psychological safety that teams need to talk about security issues, and take a user-centric approach to your processes and policies, using automation to make it easy for teams to do the right thing.
And take a look at Tanzu Advanced Edition, which gives you the building blocks needed to embed security into your pipelines and platforms. In the meantime, the amazing team at VMware Tanzu Labs is here to help transform the way your teams approach these hard technical problems and set them up for long-term success.
