Tools Resources

What Is CI/CD Security & Why Does It Matter?

Panoptica Team

Thursday, Aug 25th, 2022

Wondering about CI/CD security? We explain why CI/CD security is essential, how it works, the tools you need, and best practices to overcome its challenges.

What is CI/CD Security?

CI/CD security is a continuous process that seeks to identify and mitigate security weaknesses and vulnerabilities at every stage of the CI/CD pipeline using such techniques as:

Source composition analysis
Static application security testing
CI/CD access controls
Runtime security

CI/CD are acronyms that stand for Continuous Integration (CI) and Continuous Delivery (CD). It is a modern software practice with roots in Agile methodology and is foundational to DevOps practices that blur the lines between IT operations and software development.

CI/CD is a vital part of modern software environments but has also become an attractive attack surface targeted by malign actors. This is because they are both a conduit and repository of some of the assets and secrets critical to an organization’s enterprise.

As its name implies, CI/CD security aims to fortify software development and delivery pipelines by establishing security best practices early and continuously in the process, a growing trend commonly referred to as “shifting left.”

CI/CD security fortifies the multistage process of delivering incremental code frequently and reliably with verification of the code's correctness through automated tests and builds.

CI/CD security also reflects a step up for enterprises, as they move from separate DevOps and Security team functionality to DevSecOps approach by embracing a shift-left model that builds security from the onset of a project, through one team. This creates a CI/CD pipeline where safeguards and security checks are integrated into each key stage of the workflow.

Security testing: This typically involves all manner of automated and performance tests like unit testing, and regression testing, to detect vulnerabilities like insecure configurations and injection attacks in application builds before they are deployed to production.
Source code scanning and analysis: This allows DevSecOps to nip the potential for vulnerabilities in the bud ⁠— when developers are writing an application’s source code. This typically involves using static application security testing, or SAST is used to expose vulnerabilities in an application’s source code or binary code.

On the other hand, source composition analysis (SCA) identifies security problems, especially those lurking in open-source, third-party apps, and libraries.
Runtime security: Security testing and source code analysis are typically done during programming and pre-deployment. However, it is equally important for security testing to continue after the software has gone live to detect issues that may have arisen in production.

For instance, misconfigurations can be detected in host environments that could risk security breaches by running scans with Infrastructure-as-Code (IaC) tools and templates. Runtime security also relies on network metrics, plus application and audit logs.

The Goals and Best Practices of CI/CD Security

The importance of CI/CD pipelines has grown, especially as a vital part of developing and deploying cloud-native applications. This increased profile also heightens the need to protect CI/CD pipelines with security best practices, while keeping its security goals top of mind.

Vigilant Security Throughout All Phases of CI/CD Operations

CI/CD pipelines consist of various components that work harmoniously to achieve efficient integration and deployment of software artifacts. They typically include code and image repositories, containers, third-party tools, along with build servers.

The system hinges on various dependencies and configurations but which also make it harder to easily discern vulnerabilities. What’s more, these dependencies make it easier for attackers to find a foothold to exploit and subvert the entire system.

One of the objectives of CI/CD security is to prevent DevOps teams from falling under the illusion of seemingly secure resources. For instance, containers may seem to provide a secure environment for running applications, given their security boundaries and isolated runtime packaging built on separation of concerns. However, you still need to provide active protection for containers while they're running.

In the same manner, dependency scanning should be used to constantly analyze dependencies in the CI/CD pipeline for known vulnerabilities.

Implementing a Shift Left Strategy in a Meaningful Way

The primary goal of CI/CD security is to promote security as an integral part of the software development life cycle (SDLC) through the active engagement of DevSecOps teams. Security must address vulnerabilities in a manner that keeps up with the fast pace of other CI/DC operations.

Instead of implementing security as a final step, shifting security left entails incorporating security both early and through every layer of the CI/CD pipeline. While it aims to provide accurate threat detection at every phase, shifting left also requires both a paradigm and cultural shift by security and development teams to succeed.

Boost Productivity While Reducing Threats with the Help of Automation

Modern CI/CD practice requires software delivery to be achieved through an integrated automation process. This automation both enhances productivity and security, primarily by eliminating manual processes that are usually error-prone.

Identifying CI/CD Security Threats

CI/CD pipelines are becoming a prime target of attack because they both contain and serve as a conduit for a company’s proprietary code and sensitive information. Accordingly, adequately securing the CI/CD pipelines to prevent disruption and data leaks should be of utmost priority.

These are some of the most common CI/CD security threats but by no means represent an exhaustive list.

Insecure Code

CI/CD evolved to facilitate rapid app development lifecycles. This reduced release timeline often necessitates the use and integration of third-party and open source components into an application. What’s more, the haste and need for fast-paced delivery cycles often mean these third-party integrations are introduced into CI/CD environments without adequate scanning for vulnerabilities.

Not surprisingly, poorly vetted integrations inevitably introduce security gaps into CI/CD pipelines. These allow vulnerabilities that serve as well-known attack vectors as buffer overflows, code injection, and information leakage to occur. Moreover, these insecure code problems are exacerbated when teams fail to adhere to best practices.

Recommendations

Use a reputable code analysis tool to scan for potential loopholes and maintain your CI/CD pipeline at a high-security level. This could include source composition analysis, or SCA to identify vulnerabilities in third-party libraries.

Insufficient Pipeline-Based Access Controls

Role-based access control (RBAC) systems ensure users don’t abuse their access, maintaining compliance through the implementation of least privileges. Likewise, pipeline-based access controls perform similar functions in CI/CD pipelines.

Pipeline-based access controls are also synonymous with CI/CD access control, although the former is gaining more widespread usage.

Pipeline-based access controls exert control over pipeline execution by granting or denying access to resources both inside and outside the execution environment. Pipeline-based access controls strive to limit each pipeline to the specific set of data and resources it needs in order to operate. It is geared to restrict each pipeline step so that even when a malicious code is successfully executed, the extent of damage is isolated.

On the other hand, insufficient pipeline-based access controls increase the risk that pipeline nodes can be abused and their permissions maliciously leveraged to move laterally both inside or outside the CI/CD.

Recommendations

Ensure separate context exists for installation scripts as part of their package installation.
Do not provide installation scripts access to sensitive resources such as secrets, whether during installation or any other part of the build phase.
Apply network segmentation in such a way that it allows execution nodes to access only required and limited resources.
Scope secrets in such a way that exposes each pipeline step to access only its required secrets.
Refrain from using shared nodes for pipelines that require different levels of resources or possess varying levels of sensitivity.
Only use shared nodes for pipelines with identical roles or similar levels of confidentiality.

Poisoned Pipeline Execution (PPE)

One of the more recently discovered attack methods used by hackers against CI/CD is poisoned pipeline execution (PPE).

PPE ingeniously involves using code injection and execution to create a clear pathway to the production system. By essentially poisoning the CI/CD pipeline and build configuration, attackers have free rein to execute malicious code as part of the build process.

In other words, this malicious code is executed all within the context of the pipeline’s identity.

The most acute danger of the PPE attack vectors is that they can easily spread from simple product delivery mechanisms to wider supply chain attacks, which are very difficult to contain.

Recommendations

Remove permissions from users on the software configuration management (SCM) repositories that do not require them.
Pipeline credentials should operate on the principle of least privileges.
Any pipeline running unreviewed code should be executed on isolated nodes that lack sensitive data.
Either manage CI configuration files in a branch separate from those containing code that’s built into the pipeline or review each CI configuration file before the pipeline runs.

The Challenges with Implementing CI/CD Security

Keeping Secrets Secret in a CI/CD Pipeline

DevOps use automation tools to streamline CI/CD pipeline operations, especially during the build, test, and deployment phases. While CI/CD has traditionally dealt with application code, there is a growing shift to deploy IaC practices to build out infrastructure and manage configuration files.

However, using CI/CD pipelines to build out infrastructure increases the risk of exposing the secrets your application needs for application credentials, API keys, encryption keys, digital certificates, and so on.

While automating core aspects of the CI/CD pipeline like the test, build, and deployment tasks have the added advantage of removing the need for DevOps to store and keep track of secrets, it nonetheless moves those secrets into the CI/CD pipeline domain, along with its associated tools.

This, therefore, creates a scenario where CI/CD tools have become the biggest consumers of secrets, creating an additional challenge of storing, transmitting, and auditing those secrets securely. Moreover, when many of these CI/CD tools, by necessity, have to interface with other systems in the DevOps environment.

In other words, all these aspects together create conditions that drastically increase the risk of exposing secrets in CI/CD config files. Compromised secrets in the CD/CI pipeline allow an adversary to leak information or make unwanted changes.

Remediation Steps

Deploy the least privilege principle by granting access to only required secrets.
Ensure hardcoded secrets are removed from CI/CD tools and IaC templates.
Never store secrets in source code.
Audit CI/CD and automation tools to make sure secrets are not inadvertently passed on during operations such as build or pull requests through the CI/CD pipeline.
If necessary, spread secrets among various automation and CI/CD tools to reduce their attack attractiveness, and the potential attack target of a single file.
Rotate passwords after each use with a password manager.

Insecure System Configuration

This results from flaws in security settings. Poor and lack of proper configuration provide attackers with “low-hanging fruit” vulnerabilities in the CI/CD pipeline.

The CI/CD pipeline security is a system that depends on robust configurations and settings at various infrastructure levels, such as the network and applications. These configurations — or lack thereof — strengthen or weaken the overall security posture of the CI/CD environment.

Remediation Steps

Create and maintain a process to audit and periodically review all CI/CD environment configurations.
Use abstraction technologies like virtualization and containers, in addition to declarative IaC templates to ensure configuration parity is maintained.
Implement continuous vulnerability scanning to detect misconfigurations.

A Lack of Integrated Security Tools

Scaling DevSecOps has proven challenging, more so as organizations grapple with rapid release timelines while managing increasingly complex software infrastructures, typically spread across hundreds of server nodes.

However, automation is one of the pillars used to bridge this gap, especially in regard to application security testing in CI/CD pipelines. Embracing automation takes away a lot of uncertainty, reducing risks, and attendant rework headaches.

Nevertheless, secure DevOps is still failing to translate as it should, and ironically, technology could be the root of the problem. This is because technological barriers can contribute to a lack of effective CI/CD security.

The key DevSecOps challenge, according to 61% of respondents in a survey report by 451 Research, is the lack of automated, integrated security tools. But this lack of automation isn’t indicative of a lack of tools in the market, but rather a problem of fit.

As the report states, “not all security tools are equal, and the less software testing tools can be integrated and automated into enterprise workflows, the less effective they will be in securing CI/CD pipelines.”

Recommendations

One way organizations can overcome this challenge is to adequately identify the right type of technology that can smartly and seamlessly fit into their existing CI/CD workflows. This entails seeking out the proper approach, with the right CI/CD tools, along with the ideal vendor to provide adequate support, while continually applying industry best practices.

Securing Your CI/CD Pipeline with Panoptica

To achieve a resilient CI/CD pipeline and robust cloud security posture requires a partner who understands how to overcome potential attack vectors targeting these areas, especially within the context of your environment.

Panoptica’s context-based cloud security empowers cloud engineers and security teams to eliminate risks and maximize productivity. Panoptica’s IaC Security solution powers DevSecOps teams to ensure their code is secure from the start. Review your code as you build and garner insights into how you may be impacting production.

Try our free demo and discover the immediate positive impact that Panoptica can have on your environment from Day 1.

Get Started

Solutions

Tools

Resources

Get Started

Solutions

Tags

What Is CI/CD Security & Why Does It Matter?

Panoptica Team

What is CI/CD Security?

The Goals and Best Practices of CI/CD Security

Vigilant Security Throughout All Phases of CI/CD Operations

Implementing a Shift Left Strategy in a Meaningful Way

Boost Productivity While Reducing Threats with the Help of Automation

Identifying CI/CD Security Threats

Insecure Code

Recommendations

Insufficient Pipeline-Based Access Controls

Recommendations

Poisoned Pipeline Execution (PPE)

Recommendations

The Challenges with Implementing CI/CD Security

Keeping Secrets Secret in a CI/CD Pipeline

Remediation Steps

Insecure System Configuration

Remediation Steps

A Lack of Integrated Security Tools

Recommendations

Securing Your CI/CD Pipeline with Panoptica

Share

Tags