This blog post was written in collaboration with Emily Foster, Technical Program Manager, at Pivotal.
At Pivotal, we work with some of the world’s largest businesses in highly-regulated industries that require strict data protection and compliance. From developers to operators and executives to individual contributors, everyone in these organizations needs to recognize and adapt to changing security needs to keep things safe and secure.
This is often easier said than done. Let’s consider how great products are made: a strong and balanced product team comprised of varying functions considers the same problem with differing expertise (engineers, product, design). This push and pull is what leads to great software creation. Similarly, it takes varying perspectives of product security (product, engineering, design, information management, compliance, sales, security researchers, etc.) to create a balanced and resilient security posture.
However, this breadth of teams, while sharing a diversity of opinions, stances, and backgrounds, have historically not gotten along so well and often operate in silos. This lack of inclusivity not only hinders progress and collaboration, but also lessens the overall security posture of the product and the entire organization.
To make matters worse, security reviewers, the teams that understand how to interpret security standards and risk, are often separated from balanced product teams in different parts of the organization. This all means that these two teams don’t interact often, usually only at the intersection of various product stages, and with very little empathy for what the other does.
So how do we strive towards this balanced security team and achieve great (and secure) experiences for all?
Empathy, Diversity, and Inclusion in Security
One of the most important considerations in building a great product is the culture of the people creating it. Fostering empathy, diversity, and inclusion in your company’s culture guarantees more well-rounded and stronger products. The same goes for security, including the diversity of thought and practice in the different security roles and disciplines.
The variety of perspectives in the security space can contribute to some notable differences between product security lifecycles and common agile processes. Reviews and audits can take quite a bit of time to complete and usually need to match a specific version or build of a product to be considered valid. Product teams can provide fixes and updates iteratively, but if security isn’t involved early and often major delays and waterfall processes occur.
Additionally, standards and regulations are not often exactly written in the most user-friendly way. Security white papers, government regulations, and industry standards are numerous, and rife with security jargon that can make them quite inaccessible by “non-security personnel.” Even their formats (paper or PDF) make them inaccessible to humans and difficult to scan or search.
Let’s take a break and get to know our security and compliance teams. These teams are not only responsible for the networking, hardware, software, and physical security and processes for employees and equipment, but also for ensuring the entire company complies with complicated state, federal, and international regulatory and compliance standards that always seem to be written in large manuals with incredibly tiny print… They manage audits and are often responsible for revenue or equity loss tied to these audits. They can lose their jobs and their own personal stake in the company if they or others make a single mistake. Not to mention that there are usually multiple standards or audits to think about simultaneously, like HIPAA, FISMA, Sarbanes-Oxley, NIST, SANS, GDPR, FERPA, SOC, and others.
The world these teams operate in can be binary by nature. Organizations succeed when they pass an audit and fail when they don’t; products either meet or don’t meet standards. These definitions of success are predefined by governments or standards bodies and either non-negotiable or incredibly difficult to change. These teams operate in a trust but verify model when they not only need to meet standards, but they need to have proof of their compliance.
This is quite different from the way that balanced product teams generally think about product requirements. Product teams can compromise, reprioritize, or pivot away from work. Their definition of success is customer value, and shipping a third of a feature can still provide customer value. However, the commonality and important thing to remember with all of these groups is that everyone is approaching the end user and product in the same way: they want to create a safe product and enjoyable experience for everyone. This experience is the ultimate definition of value that matters.
How To Do Product Security Better
We know that empathy, diversity, and inclusion are all key parts to a successful culture and therefore product (hello Conway’s Law). And we see the diversity of thought and practice within various security roles and disciplines So what kinds of changes can teams and organizations make to foster these values and maximize product security outcomes?
Product Teams
One of the highest value changes product teams can make is to incorporate the security personas we describe here into your usual product personas. Start treating internal auditors and compliance teams as users and understand their definition of product value. Taking these personas into account earlier might cause a product team to build in functionality that will help with security reviews and audits and save significant time and pain later on.
Consider using standardized basic libraries across teams to speed up the audit process and allow security teams to focus on net-new features and potential vulnerabilities. This is also an excellent opportunity for pairing to learn about security development lifecycle practices such as threat modeling and security testing.
It can also be helpful to know what is generally expected by certain standards and certifications. For instance, if you’re a global organization that stores data or builds products that store data, having a basic understanding of GDPR will immensely help your customers and users. Rather than falling victim to the usual waterfall nature of a security process where groups provide requirements to product teams later in the product lifecycle, teams should be treated as any other member and be involved in discovery and framing, inceptions, etc. At minimum, teams can invite security groups to your internal demos early on. This will encourage communication and education at the point in your product lifecycle when change is cheap, which makes everyone happy.
Finally, consider this an opportunity to gently educate others and let them know that you’re considering their safety and security when designing your product. A safe experience is a good experience!
Information Security and Compliance Teams
Security teams can start to align their processes with those of modern agile product teams. It’s all about education and empathy here. Currently, there are still not many compliance and security tools that play well with development pipelines; DevOps and security teams are still figuring out how to work best together. As our Global CTO of Security and Compliance, David Zendzian, notes, it’s important to remember that product teams are incentivized to deliver business value not security. Security teams could frame their work more in terms of business value rather than avoided risk. Building in logging capabilities, patching vulnerabilities quickly, making it easier to detect breaches, and protecting data is as much a security function as closing gaps in an audit checklist.
When security gets involved too late the discussion it becomes a conversation of whether something is “worth the risk exposure,” when it could be a conversation about added value.
Similarly, security teams can get involved in the product lifecycle early on to educate, collaborate, and enable teams to continuously incorporate security considerations into their development process. Consider allocating specific team members to review specific products so they can contextualize the standards and regulations they seek to meet. Providing perspective as well as expectations for upcoming security requirements will make the audit and compliance process down the line go far more smoothly. And empathizing with product teams and reflecting their workflow with agile governance will cultivate a productive working relationship.
In the same vein, this continued contact should help foster a culture of healthy and psychologically safe communication among security groups that would otherwise be siloed. The worst case scenario for an organization when it comes to security is fear. When people are scared they don’t speak up, they don’t report small errors that could potentially be harmful, and they aren’t encouraged to take the time to do things securely. Security teams should be available for questions as opposed to pointing teams to lengthy PDFs with security jargon. As we all know the security landscape is constantly changing, and making security teams about enablement and education will allow organizations to build better products that maintain their security over time.
Executives and the Organization
When jobs are on the line it’s easy to create silos and structures that create accountability. However, boards and executives should be responsible for encouraging a blameless culture to foster communication and improve security. Leadership can help address organizational structures and values at the company that directly impact the ability for employees to do security well. Start by encouraging staff to do the right thing and make time for security early and often. Don’t let security and compliance reviews be a last minute gate to getting products out the door. Encourage security, compliance, and product teams to collaborate with product teams throughout the entire product lifecycle. Sit them next to each other or let folks rotate between teams so they can empathize and include one another.
Another approach might be to create boards comprised of cross-functioning individuals and SME’s that consider security broadly across the organization. They can facilitate conversations, be responsible for finding and enabling best practices, and mandate healthy behaviors to keep these processes fresh. A cross-functional board like this might also help bring issues to the attention of executives since they understand the holistic nature of the organization's security.
In Conclusion
Security is not going to get any easier or slow down as software continues to become a larger part of the world. We know that there are some similarities but also a few significant differences between a product lifecycle and a security lifecycle that make it difficult for these roles and disciplines to interact. We also know that empathy and inclusion of differing experiences begets better products, and in our case more secure products. Everyone has the same outcome in mind: a delightful, safe experience for users. The more that teams can recognize their common ground and work together when it comes to security the safer we will all be.
Shout out to David Zendzian and Coby Almond for their help reviewing.
The Players Involved
Product security starts with the core of a balanced product team that focus on their customers and users. When these balanced teams also consider concepts like identity management, data protection, or stability, they ensure their product is secure for their users. But teams also need to consider the other players involved in the security space within their organization.
Information Security
Information security and technology teams maintain tooling and monitor information to ensure that product teams operate securely and effectively. Working at this scale can be a daunting task. These teams often work not only with software but also hardware and physical security. Security teams might also run exercises such as threat modeling to assess the product’s secure design and potential for abuse. While compliance usually makes the policies, InfoSec usually focuses more on procedures related to both business and software development.
Compliance
There is also a need to ensure that the product and organization are in compliance with relevant regulations and standards. This role could be fulfilled by the same information security team, a separate compliance team, or an external firm. Product organizations need to be able to prove their compliance to external auditors or regulatory agencies somehow – auditors cannot simply trust our word, they must also verify.
Executives
Each of the groups described above have executives with varying and at times competing interests to balance security with performance or other organizational needs. Much like a balanced product team considers the problem from various viewpoints, this executive board wants to understand the overall security posture of the entire product and how it’s being used in their company’s context. Having a CISO or someone with security expertise can prove valuable to help navigate the complexities and changing security landscapes.
