Summary: The DevOps movement has evolved the way on how the organization operates and DevSecOps a new practice came along quickly by shifting security a lot to the “Left” most point of the software development lifecycle. Meaning, Security has to be embedded (not bolted) in all phases of the software lifecycle, starting from the Planning stage to the Requirement stage to Design to Code, through Test, Release, and Monitoring. This article explores what is shift left in the DevOps world and key considerations of DevSecOps that have to follow to integrate security into the DevOps process.
BottomLine: Moving Security to Left will help to identify and fix security issues at an earlier stage which will eliminate all the risks associated with the delivery process and eventually result in building and running secure systems.
What is the difference between SecDevOps vs DevSecOps vs DevOpsSec?
Despite calling SecDevOps, DevSecOps, DevOpsSec, or Rugged DevOps, the following words mean Integrating security practices in DevOps Lifecycle. Refer here for the difference.
Benefits of Shift Left in DevOps Lifecycle
The following is the list of summary of what can be achieved by shifting the security to the left as part of the DevOps lifecycle.
- Continuous Software Delivery – Early identification and fixing of vulnerabilities with code in the development phase.
- Less complex problems to fix at a later point in time.
- Quality – Can be achieved by automating the security testing and security rules in the CI/CD pipeline.
- Secure Environment – Can be achieved by continuously scanning your environment and tear down in case of malware attacks.
- ‘DevSecOps’ purpose is to make ‘everyone is responsible for security’.
Traditional Security vs Shift Left Security
Traditional Security: Most of the organizations were handling security with separate roles, outside of the development cycle. ie Security is often left as an add-on or a gated process that is practiced randomly before the delivery process.
Bottom line: It’s always difficult to fix security issues before release and usually nullify all benefits and result in error-prone application delivery to customers and it is not a scalable option and adds delay to the development cycle as well.
DevSecOps: In the case of DevSecOps, security should be part of the process, and developers are key roles who will be taking responsibility to build and automate the security process and tools in the entire Product workflow.
The focus of the security team in this new paradigm is to collaborate with developers, empowered them with required security tools, train them to address vulnerabilities, enabled them to use required tools, and security best practices.
Bottom line: The security practices will be owned by all teams in a release candidate and strictly it is not just only the security team. Also please note, an organization will fail in this process if outdated security practices are followed.
Why security is important?
Today most of the security breaches are due to application vulnerabilities. Refer the below stats will give you a glimpse of few security breaches that grabbed headlines in 2019. As per the study the estimate is about $3.92 million (Link). Also, it is a significant increase of 1.5 percent compared to last year (2018). Source
- First American Corporation: (Financial service company): 885 Million Records Exposed Online: Bank Transactions, Social Security Numbers. (Link)
- Facebook (social network): Hundreds of millions of Facebook user records were exposed on the Amazon cloud server. (Link)
- Truecaller (Telephone directory): Data from caller identity app Truecaller, including names, phone numbers and email addresses of users worldwide, is available for sale on private internet fora, according to a cybersecurity analyst who monitors such transactions. (Link)
- Facebook (Social Network): Millions of Facebook user phone numbers exposed online, security researchers say. (Link) and Over 267 Million Facebook Users Have Account Info Exposed On Dark Web In Massive Data Breach. (Link)
Key considerations to implement DevSecOps.
It is worth thinking about the key considerations in DevSecOps that should be satisfied before implementing a CICD pipeline of your applications.
i. Automate security
The first step in the DevOps process is to automate security from day one. It not only makes the whole process repeatable and faster, but it fits with the way development teams work. By automating security, teams can ensure that each component gets all the security testing it needs without taking up any extra resources, making security a part of the development process itself.
The key to automating security is to look at the tools you already have in place. Identify the ones you use for continuous integration, continuous deployment, bug tracking, and more to determine the best points to introduce automation. The more you can automate to make it a part of the developers’ process, the less work a security team will need to do later point of time.
ii. Shift Left
Next, integrate security into the CI/CD process as early in the development lifecycle as possible. Your goal should be to minimize the gap between the discovery of a problem and the time it takes to bring the developer back in to fix it.
That’s because it’s much faster, cheaper and easier to ask a developer to fix something they just coded compared to something they wrote six months ago. The longer you wait, the longer it will take for the developer to get back up to speed with that particular code, assuming the developer is even still on the project.
The goal is to conduct security testing during continuous integration. Better yet, use automation to conduct testing while the developer is writing the code, giving them instant feedback so they can make a fix before it ever becomes a problem.
iii. No False Positives
Any test that creates too many false security triggers is as bad as one that lets flaws be overlooked. This is especially true in CI/CD, where a failed security test can stop critical business features from being delivered to production, or an important production patch being released.
That’s exactly what should happen if the security issue is real, but it’s completely unacceptable if the finding is a false positive and result in the delayed development cycle.
iv. Scale DevSecOps
Most developers aren’t trained in the practices of secure coding. By embedding application security knowledge directly in the team, you can give your security team a force multiplier while reducing culture conflict.
Start by designating one person on the development scrum team as the go-to resource for application security. When you have a security champion embedded with every scrum team, you’re guaranteed to have security represented in every design decision and at every stand-up. This helps remove the security bottleneck that can occur when you have a traditional centralized AppSec expert who’s outside of development; it also helps the security team be represented in more places at once.
Concluding
Take these above key considerations and approaches to your DevSecOps lifecycle will have better results in quicker time. DevSecOps is a better solution that organizations should strive for. However, the most important consideration when building a DevSecOps environment is to ensure you have the toolset and model.
How to shift left with continuous integration pipeline – Checklist
DevSecOps focus on multiple areas such as Dev Code Analysis, Code Change Management, Threat Investigations, Vulnerability Management and Mitigation, On-Going Training, Continuous Testing, and Compliance Monitoring.
To have the successful security integration into the CICD pipeline we have to adhere to security best practices starting from planning, coding, testing, release, and packaging, to deploying, operating, and monitoring. The below infographic will walkthrough on enabling security into your CICD pipeline with DevSecOps, this practice will differ from organization to organization.
Also, keep in mind that security automation will not find all security issues. The main goal is to catch what we can and use our Developers to find high-severity vulnerabilities
a. SourceCode
This is the first step in the DevOps pipeline, It is mainly used to find and fix common security issues before the developer committing the source code to the repository. it is typical to require that the code passes a number of checks referred below.
Controls:
- Peer-based code review
- Unit test coverage
- Static code analysis: involves statically scanning for performance, reliability, and security issues as part of the merge decision for new code.
b. Build
In this step source code will be compiled, unit tested and scanned for quality and security and below security checks and controls to be added in this step are referred below
Controls:
- Build configuration: Essentially build configurations should be source controlled and peer-reviewed just as the application code.
- Dependency management: To ensure that every dependency downloaded is whitelisted by trusted sources and scan-build dependencies. They should comply with the license, security, quality check and it should be approved GA version.
- Unit test: Every build should include unit test execution and should complete successfully only if the unit test pass rate and coverage meet predefined criteria.
- Static security analysis: In addition to unit test execution, source code also should be scanned for potential security vulnerabilities.
c. Packaging
In this stage, the deployable artifact is composed of source code, external dependencies and requires the following checks referred below
- Image signature: Validate the digital sign of Deployable artifact to ensure authenticity
- Runtime configs: Validate run time configs associated with the artifact.
- Metadata Management: Artifact should have valid metadata to identify the version and other details.
d. Artifact Repository
Version control which will have all the artifacts produced via build and packaging.
- Artifact Repository: Configure the artifact repository to accept upload requests only from known and trusted packaging sources. Individual users should be restricted from uploading to the artifact repository.
- Immutable artifact: No artifact in the repository can be overwritten; only a newer version of the same artifact can be uploaded.
- Retention policy: The artifact repository should implement a retention policy for all released artifacts.
e. System testing
The customer-facing applications should be tested with minimum peak load and mainly it will allow us to learn and benchmark the system’s behavior under both off-peak and at peak conditions.
- Soak testing: Check system with a typical production load, over a continuous availability period, to validate system behavior under production use.
- Fuzzing: Typically forcing lots of random garbage at applications, seeing whether any particular (type of) garbage causes errors and how the application is behaving.
- Chaos Engg: Introduce random failures, usually in the production environment, to see how application environments handle adverse conditions. This helps us to be prepared for adverse impacts at the time of peak and off-peak conditions.
f. Pre-Production
After code freeze, the current release candidate (application) will be deployed into the Pre-Production environment. This environment will normally hold, an exact mirror of the production environment. Stress testing will be performed and followed by business/client/manager approvals the artifacts will be pushed to the production environment. Below are the controls that need to be considered in this stage.
- Fetch artifact only from a trusted source: The non-production deployment stage should fetch deployable artifacts only from trusted sources (such as the enterprise’s artifact repository).
- Whitelist of allowed connectivity: The connections allowed in the nonproduction deployment stage should be reviewed and kept up-to-date.
- Whitelist of allowed data: The non-production deployment stage should have access to only a set of whitelisted test data. This data should not contain any real customer data or sensitive information.
- Quality gate evaluation: Non-production deployment should be executed only if it meets a set of predefined criteria (e.g., 100% test pass rate with 80% coverage, no new high severity security vulnerability, etc.). The quality gate also should consider the drift and difference between production and nonproduction environments; the non-production environment should mimic the production environment.
g. Prod Deploy
This environment will be hosted with the current released version of the application which will be served to customers (live traffic). The version will not change unless and until there is a scheduled release for new features/Hotfixes. Below are the controls that need to be considered in this stage.
- Fetch artifact only from a trusted source: The production deployment stage should fetch deployable artifacts from only trusted sources (such as the artifact repository).
- Only use approved configurations: The production system should use an approved set of configurations, such as network connectivity, encryption, tokenization, secrets management, etc.
- Security monitoring: The production system should have intrusion detection, threat monitors, and other approved security mechanisms. It should also have approved monitoring, logging, and alerting mechanisms.
- Change management: The production system should have an automated change management mechanism.
- Access control: The production system should have a strict access control mechanism. By default, no one should have access to production systems except by means of a break-glass mechanism.
- Deployment strategy enforced: Production deployment should enforce deployment strategy (e.g., deploy only during a specified time window, use blue-green deployment, use canary deployments, etc.).
OWSAP proactive controls
The Open Web Application Security Project (OWASP) Top Ten Proactive Controls is a list of security techniques that should be included in every software development project and dedicated to enabling organizations to design, develop, acquire, operate, and maintain secure software. They are ordered by order of importance, with control number 1 being the most important. This document was written by developers for developers to assist those new to secure development.
C1: Define Security Requirements
C2: Leverage Security Frameworks and Libraries
C3: Secure Database Access
C4: Encode and Escape Data
C5: Validate All Inputs
C6: Implement Digital Identity
C7: Enforce Access Controls
C8: Protect Data Everywhere
C9: Implement Security Logging and Monitoring
C10: Handle All Errors and Exceptions
For more information: All OWASP tools, documents, forums, and chapters are free and open to anyone interested in improving application security. It can be found at www.owasp.org
Also, refer to the OWASP Cheat Sheet collection which will help you to know simple good practice guides for specific application security topics. These cheat sheets were created by various application security professionals who have expertise in specific topics. Worth considering these best practices when you are designing your applications.
Conclusion
“DevSecOps” is not a thing, it’s debatable (even to me) whether it should exist as a term.
Embed security throughout the DevOps lifecycle to identify vulnerabilities earlier, perform faster fixes, and reduce cost. So Securing DevOps is a continuous journey. You can refer current leaders like Google, Amazon … for more Gyan on shift left security in DevOps.
In my experience, we have created a strong process of code reviews and implemented continuous integration with an automated test suite which helped us to target low-hanging fruit first (i.e., the most pressing vulnerabilities) and frequent iteration helped us to lower the risk with each successive release and enabled our development team to build security and get the vulnerabilities fixed by shifting left.
So start small with meaningful changes and keep iterating.
References
1. SecDevOps vs DevSecOps vs DevOpsSec 2. IT Revolution Press, LLC, 25 NW 23rd Pl, Suite 6314, Portland, OR 97210
