Protect AWS CodePipeline CI/CD workflows with SUSE Security

Guest writer: Dominik Wombacher, Sr. Partner Solutions Architect, AWS

Learn in this Blog how SUSE Security helps to protect AWS CodePipeline CI/CD workflows. This is achieved by conducting security testing within the Pipeline. And ensures that only Artifacts without vulnerabilities are promoted to the next phases. Once in production, SUSE Security continuously scans and protects your workloads across the whole lifecycle.

Risks to mitigate

The classic CI/CD Workflow is focused on building code, performing unit- and / or integration tests and promoting the resulting artifacts into production. There is nothing wrong with this approach shown in Figure 1\. But there is a crucial part missing in this model: Security comes into the picture when your code already runs in production.

Figure 1: Classic CI/CD workflow, focused on code build, testing and promotion into production.

The sample solution described in this Blog adds SUSE Security into the Build and Staging phase. This shifts the Security focus to early stages in the development and deployment workflow.

Walkthrough

Let’s examine the Architecture overview before we dive into the individual steps.

Figure 2: Example CI/CD workflow with AWS CodePipeline with SUSE Security as part of the build phases and continuous protection in production.

The Workflow shown in Figure 2:

1. Bugs are fixed or features added to the application source code
2. Code changes are committed and pushed to a git repository
3. AWS CodePipeline notice that there is a new commit
4. The latest version of the code is received in the Source stage
5. AWS CodeBuild creates a container image which is tested by SUSE Security. The image is then pushed to Amazon ECR if no vulnerabilities are found.
6. The application is deployed for further validation in the testing stage
7. After passing all tests, the application is promoted into production.
8. SUSE Security protects and scans the application through its lifecycle

Solution

The sample solution in this Blog covers the workflow steps 1 till 5\. The used Buildspec and Scan script are available in aws-samples on GitHub. Following a step by step walkthrough with screenshots and descriptions to follow along.

Figure 3: Bugs are fixed or features added to the application source code. Code changes are committed and pushed to a git repository.

The Dockerfile of the demo application in Figure 3 uses the SUSE Base Container Image (BCI) for Node.js in version 16. Npm packages are updated and a http-server installed. The server runs on Port 8080 when the container starts. The changes are committed and pushed to a git repository. The Code:

FROM registry.suse.com/bci/nodejs:16

WORKDIR /app
COPY . .

RUN npm update \-g \\
&& npm install http-server \-g

CMD \["http-server", "-o", "/app"\]
EXPOSE 8080

Figure 4: AWS CodePipeline noticed that the code changed. The latest version of the code is received in the Source stage.

After the code changes were pushed, Figure 4 shows how AWS CodePipeline polls for changes, recognizes that there is a newer commit and downloads the revision as part of the Source stage.

Figure 5: AWS CodePipeline triggers AWS CodeBuild.

After successful completion of the Source stage, AWS CodeBuild triggers the configured Build stage in Figure 5.

Figure 6: The git commit hash of the last push that contains the Dockerfile of the demo application.

Throughout the build process, the git commit hash shown in Figure 6 (6909610f) is used as the unique identifier. This allows you to identify the individual build and relation between a container image and the underlying source code.

Figure 7: Overview of the AWS CodeBuild status and phase details.

The INSTALL and PRE\_BUILD phases in Figure 7 run before the BUILD phase and prepare the environment for the actual image build and security scan. This includes login to AWS Elastic Container Registry (ECR) and the start of a local docker registry.

A newly created image is first pushed to this local registry before the SUSE Security scan is performed. This is a workaround because the SUSE Security controller expects that a submitted scan result contains a value for Digest or RepoDigest.
Otherwise the upload fails with:

ERRO|SCN|main.main: Failed to submit scan result - error=Submit scan result failed with status code 400.

When an image is built locally inside AWS CodeBuild via docker, then this value is missing. This is per design and how docker works in general. The value is generated and set as soon an image is pushed to a registry.

Until this behavior is changed in SUSE Security, this additional step is necessary to submit scan results.

Figure 8: AWS CodeBuild logs that contain information about the created image, the applied tag and the SUSE Security scan results.

AWS CodeBuild creates a container image, tagged with the git commit hash, which is then scanned by SUSE Security. The results in this example in Figure 8 are: 72 vulnerabilities in total, 31 high, 25 medium and 16 low.

Figure 9: AWS CodeBuild logs with scan results of npm modules inside the container image.

In addition to packages of the container Base image, code related packages, for example npm, were scanned as well. Figure 9 highlights that a medium severity CVE in the tar package was identified. The scan report was then successfully submitted to SUSE Security.

Figure 10: AWS CodeBuild logs with build status FAILED

Because of the found vulnerabilities, the build stage in Figure 10 failed and the container image wasn’t pushed to the registry. The developer needs to fix the issue and re-submit the code without high severity vulnerabilities to pass the build stage.

Figure 11: AWS CodePipeline shows the build as failed.

In addition to the status in the AWS CodeBuild logs, AWS CodePipeline also shows the Build as failed in Figure 11. The git commit hash confirms that this is related to the latest update.

Figure 12: Risk Reports overview in SUSE Security with details about the failed build.

The value of sending scan results back to SUSE Security is the accessibility at a later point. Developer and Stakeholder can search for git commit hashes of a specific build and access the detailed report as shown in Figure 12. This allows an analysis of the identified vulnerabilities and decisions on how to fix them. It can also be used as a long-term archive for compliance and auditing purposes.

Figure 13: Code Editor to perform changes on the demo app and push them to a git repository.

To fix the found vulnerabilities, the base image has to be changed to the latest tag as shown in Figure 13. Git commit and push the latest changes to a repository triggers AWS CodePipeline.

Figure 14: AWS CodePipeline noticed that the code changed. The latest version of the code is received in the Source stage.

As in the first iteration, Figure 14 shows that AWS CodePipeline polls for changes and receives the code from the latest commit in the Source stage.

Figure 15: AWS CodePipeline triggers AWS CodeBuild.

A new Build is triggered and Figure 15 shows the new git commit hash 9d911191 that will be the unique identifier for this run.

Figure 16: AWS CodeBuild log output with a container image that has the new git commit hash as tag and the scan results from SUSE Security.

AWS CodeBuild creates again a container image and initiates the scan by SUSE Security. This time the result is 1 vulnerability in total, severity medium. Figure 16 also shows that the scan results are successfully submitted back to SUSE Security.

Figure 17: AWS CodeBuild logs with a successful Build state.

The used scan script is configured to fail if high severity vulnerabilities are detected. The current scan only contained 1 medium severity finding. Figure 17 shows therefore that the build is flagged as successful and the container image pushed to AWS ECR.

Figure 18: Validation, the image is tagged with latest and the git commit hash in AWS ECR.

The ECR repository in Figure 18 shows that the latest image is also tagged with the identical git commit hash that was used to trigger the build.

Figure 19: Risk Report returns the details of the scanned container based on a search for the git commit hash.

The final verification step in Figure 19 is to search for the scan report in SUSE Security based on the git commit hash. It aligns with the result of the AWS CodeBuild stage, 1 medium severity finding.

Figure 20: Extract of the Buildspec that shows an example how the environment variables NV_CONTROLLER and NV_PASSWORD are configured.

The necessary environmental variables are configured in the Buildspec as shown in Figure 20. It’s important to highlight the NV_PASSWORD variable. This value is used to authenticate against the SUSE Security API. To avoid clear text credentials, it is recommended to add the password to AWS Secrets Manager. This example Buildspec tells AWS CodeBuild to search for a secret named neuvector-admin-password and then inject it automatically into the build environment.

Conclusion

This Blog demonstrated how SUSE Security helps to protect AWS CodePipeline CI/CD workflows. The sample solution uses the Buildspec and Scan script available in the aws-samples project NeuVector vulnerability scanner in AWS CodeBuild on GitHub.

It’s your turn, time to integrate SUSE Security into your CI/CD workflow by yourself! Use the sample solution as a blueprint and adapt it based on your own needs and use-case. The commands in the Buildspec and the scan script are not tightly coupled to AWS CodePipeline and can be used as a starting point for other CI/CD systems as well.

Further reading

SUSECON 24: [TUTORIAL-1156] NeuVector Integration into AWS CodePipeline CI/CD Workflow
openSUSE Conference 24: NeuVector Integration into AWS CodePipeline CI/CD workflow
SUSE Security – Standalone Scanner for Local Scanning
SUSE Security – Build Phase Image Scanning
AWS CDK Workshop
EKS Blueprints for CDK Workshop

Author

Dominik works as Sr. Partner Solutions Architect, with a focus on SUSE products, in the Linux Partner Team at AWS. He is an Open Source Enthusiast and Contributor, Dog Person, Passionate Engineer who loves solving tricky issues and is always eager to learn new things. His professional career started in 2002 and has always been IT-centric, distinguished by broad knowledge of different technologies and fields. At AWS, he helps Partners and Customers to optimize existing and to migrate new workloads to AWS.