.png)
DevOps is not only tooling and technology – much of its success lies in how people work within the model. This post aims to guide security peers outlining our role in a DevOps environment, while minimising friction.
From own experience, Dev and Ops teams are likely to think security only becomes part of the operational process of integrating and delivering code. I call Security out as a separate item because, even woven into the DevOps framework, it’s substantially more difficult for a security expert to fit in. We need to consider how we can improve delivery of secure code without waste and without introducing bottlenecks in a development process which we may not be intimately familiar with. The good news is that security fits nicely within a DevOps model, but we do need to tailor things to work within automation and orchestration to be successful.
Cloud and DevOps go hand in hand. "DevOps-First" approach is the idea of not only doing the Cloud, but also doing it right, by utilising IaC, CI/CD, Cloud-native services and SRE principles.
I'll try to detail few key areas where we can integrate security with a DevOps model:
Learn the DevOps model: To work in a DevOps environment we need to understand what it is and how it works. We need to understand cultural and philosophical changes, as well as how they affect process, tooling, and priorities. We need to understand the organisation’s approach to optimally integrate security tooling and metrics. Once we understand the mechanics of the development team, we'll have a better idea of how to work with them in their own process.
Learn how to be agile: Our participation in a DevOps team means we need to fit into DevOps – not the other way around. The goal of DevOps is fast, faster, fastest: small iterative changes which offer quick feedback, ultimately reducing Work In Progress. Small, iterative changes to improve security fit this model. Prioritise things which accelerate delivery of secure software over delivery of new features – keeping in mind that it is a huge undertaking to change culture away from “feature first”. We need to adjust requirements and recommendations, so they fit into the process, often simplified into small steps, with enough information for tasks to be both automated and monitored. We can recommend manual code reviews or fuzz testing, so long as we understand where they fit within the process, and what can – and cannot – gate releases.
Educate: One of the best ways to bring a development team up to speed in security is training: in-house explanations or demonstrations, 3rd parties' experts to help with application security or threat-modelling, eLearning, or various commercial courses. The historical downside has been cost, with many classes costing thousands of dollars. We'll need to evaluate how best to use resources – the answer typically includes some eLearning for all employees, and select people attending classes and then teaching peers. On-site experts can be expensive, but an entire group can participate in training.
Grow our own support: There is simply no way for security teams to scale out knowledge without help. This does not mean hundreds of new security employees – it means deputising developers to help with product security. Security teams are typically small and outnumbered by developers 100:1. What’s more, security people are not present at most development meetings, so they lack visibility into day-to-day agile activities. To help extend the reach of the security team, we should try to get a Dev team member – a “security champion” – to act as a security advocate. This helps not only extend the security team’s reach, but also expand security awareness across.
Help DevOps team understand threats: Most developers don’t fully grasp how attackers approach systems, or what it means when a code or SQL injection attack is possible. The depth and breadth of security threats is outside their experience and most organisations do not teach threat-modelling. The OWASP Top Ten is a good guide to the types of code deficiencies which plague Dev teams, but we should map these threats back to real-world examples, show the damage that a SQL injection attack can cause, and explain how a 'Heartbleed' type vulnerability can completely expose customer credentials. Leverage these real-world use cases as “shock and awe” to realise it to the developers.
Advise on remediation practices: Our security programme is inadequate if it simply says to “encrypt data” or “install WAF”. All too often, developers and IT have a singular idea of what constitutes security, centred on a single tool they want to set and forget. Help build out the elements of the security programme, including both in-code enhancements and supporting tools. Teach how those each help to address specific threats, and offer help with deployment and policy setup. In the past, organisations used to produce code style guides to teach junior developers what properly written code looked like. Typically these are not online. Consider a wiki page for security coding practices, or engage with the appropriate SME team, if exists.
Help evaluate security tools: It is unusual for people outside security to fully understand what security tools do or how they work. WE can help in two ways; first, help developers select tools. Misconceptions are flourishing, and not just because vendors over-promise. Additionally it is uncommon for developers to evaluate code scanners, activity monitors, or even patch management system effectiveness. In our role as advisers it's our responsibility to help DevOps understand what the tools can provide and what fits within our shared testing framework. Goal is to know when a product fails to deliver meaningful results, as well as help to position the expenditure, as it’s not always clear to budget holders how specific tools address security and compliance requirements. We should specify functional and reporting requirements for tools which meet business needs.
Help with priorities: It's somewhat challenging to differentiate high priority vulnerabilities with impact on the organisation, from lower level ones. Risk Analysis is outside the developer skill set. We need to help fill this gap because not every vulnerability poses residual risk. It's not enough only to raise the risk, but we need to be able to articulate it by mapping a threat to possible exploitation, what the exploit might mean to the Business, or what can been done to address and reduce the risks. For example, we might be able to remediate a critical application vulnerability in code, patch supporting systems, disable the feature if it’s not critical, block with IDS or firewalls, or even filter with WAF or RASP technologies. Sometimes code exploitation cannot actually harm the business, so “do nothing” is the right answer! There are typically several options to address a vulnerability, so presenting trade-offs to a DevOps team allows them to select the best fit.
Write tests: DevOps has placed some operations and release management personnel in the uncomfortable position of having to learn to script, code, and expose their work for public review. It pushes people outside their comfort zones in the short term, but that is a key part of building a cohesive team in the medium term. It is perfectly acceptable for Security to contribute tests to teams (e.g. scans to validate certificates, checks for known SQL injection attacks, open source tools for locating vulnerabilities, etc.). To participate in a DevOps team, where automation plays a key part, we will likely need to know how to write/read scripts or templates, with a great benefit that security policies will be embodied into the definition of the environment from Build stage.
Advocacy: One crucial area where we can help development, is in understanding both corporate and external security Policy requirements. We're being seen as the link to those Policies/Standards. We must understand basics of compliance, privacy, software licensing and security Policies, to be able to describe back the requirements and their rational.
We, the security teams, are being tasked with getting security into DevOps, and sometimes struggling to integrate, culturally or technically, into DevOps, continuously been asking questions and assistance. Getting the attention of developers, much less their participation, is a real-world problem today. The challenge of security testing, and the problem of security controls being outside the domain of developers and IT Ops, can be mitigated with DevOps. Security can be largely automated, built slowly over time, and part of the entire release process, with Security embedded in the delivery process ("shift left").
It is expected for this journey not to be easy, as the path from traditional development to DevOps and DevSevOps requires a significant amount of work, while abandoning old methodologies, but with great benefits which will be applied across teams' functions and the Business as a whole.
Comments