Dive into how SUSE® Rancher Prime automation fixes CVEs to fortify security in containers and dependencies

The amount of vulnerabilities, aka CVEs (Common Vulnerabilities and Exposures), affect all forms of software from is grows exponentially. New CVEs are discovered every day, affecting both old and newly released software. Consumers expect companies to address and fix CVEs promptly, but this process is often more complex and time-consuming than anticipated meaning that dependencies can leave you at risk. If you are dedicated to security and want to know how SUSE®security experts fix and prevent CVE’s and the continuous security improvements we ship so you can improve your posture read onward. 

Additionally, many security tools and CVE scanners rely only on package version identification to flag vulnerabilities. These scanners often fail to recognize whether the affected code paths are in use, leading to false positives. For example, a dependency might have a CVE, but if the affected code path isn’t used, the software isn’t vulnerable. This high signal-to-noise ratio is a significant challenge for developers and operations teams trying to maintain secure environments.This generates a high CVE signal to noise ratio in some software when the scanners flag CVEs that are actually false-positives 1.

How SUSE® engineering teams battle CVEs

To address the constant influx of new CVEs and to provide more secure software to our consumers with reliable CVE data, the SUSE® Rancher Security team is continuously implementing tooling, automation and best practices to support our Engineering teams in prioritizing and fixing the applicable CVEs. Examples of such measures are:

1. Daily Automated Scans Across All Images Scanning of CVEs across all images that are shipped in SUSE® Rancher Prime – daily automated scans are executed to scan all images that are currently shipped in SUSE® Rancher Prime, Harvester, Longhorn and NeuVector2.
   1. These scans are capable of identifying CVEs in direct and indirect dependencies that we use to build our software (binaries) as well as in OS level packages inside the container images.
2. Manual CVE Triage daily review of CVEs that are identified with the automation cited before, as well as those that are manually identified or reported to our Security team.
3. Source code scanning daily automated scans to identify new vulnerabilities and to remove false-positives directly at the source code level.
4. Automated Dependency Bumping tools like Dependabot, Renovate and Updatecli are constantly being rolled out across our repositories to help teams automate the update process to fix CVEs with the bumping of the dependencies that we use. This process is not only about security, but a good dependency hygiene measure to help keep the dependencies stable and updated with their upstream releases.
5. Monthly patch releases the current SUSE® Rancher Prime release cycle decreases the time users need to wait for dependency bumps, security fixes and improvements. For more information, please check KB 000021405.
6. Antivirus scanning similar to the CVE scanning process, a weekly automated antivirus scan is executed against all the images that ship in SUSE® Rancher Prime, Harvester, Longhorn and NeuVector2.
   1. If you believe that you have found a virus in one of our software (images or binaries), please consult KB 000021432 about how to proceed.
7. Software Bill of Materials (SBOM) and Provenance rolling out the embedding of SBOM (software bill of materials) and SLSA (Supply-chain Levels for Software Artifacts) provenance manifests as layers across the container images that are part of SUSE® Rancher Prime.

Full transparency: Communicating CVEs to customers

All CVEs that directly affect our products at the application layer, e.g., the CVEs that are filed directly against SUSE® Rancher Prime, RKE2, Harvester etc. as the source of the vulnerability, are made available in the source code repository of the affected product. For example, see SUSE® Rancher Prime’s main repository in GitHub for its CVEs or RKE2’s main repository. They are also listed in the respective release notes of the version that contains the fix to the CVE and also in SUSE® Rancher Prime’s security advisories page.

Since June 2024, we are also publishing the list of CVEs affecting our dependencies and container images in SUSE’s CVE database page. There, one can search by CVE number and check all SUSE’s products that are affected by that CVE, including images shipped in SUSE® Rancher Prime, RKE2, Harvester, etc. 

At the moment, we only publish information for critical and high severity CVEs in images that are directly built by us, such as the main rancher/rancher image, rancher/rancher-agent, rancher/rke2-runtime, rancher/fleet, rancher/harvester and others. We are working on expanding the scope for the published information, which in the future will include upstream and mirrored images (those that are built by third-party projects).

Reducing CVE false-positive noise

To help fight the CVE fatigue problem and to reduce the false-positive CVE noise in our images, the SUSE® Rancher Prime Security team just started publishing VEX reports in our VEX Hub repository. VEX, which stands for Vulnerability-Exploitability eXchange, is an open specification, developed by NTIA3, to report if a certain CVE is applicable or is a false-positive on a given version of a software.

For now, our VEX reports will contain only the confirmed false-positives CVEs in our products. Please consult KB 000021573 for more information about VEX and how to use the reports to remove false-positives when doing your own scans against our products.

How fast we fix CVEs

Patching CVEs across a large ecosystem isn’t a trivial task. First, the CVE might not even be applicable to the code base (that is why we are releasing our VEX reports now). Second, even if the CVE is applicable, it might not be easily exploitable (VEX will help us here in the future to correctly communicate the impact of a CVE). Third, a stable patch might not be quickly available. And fourth, when a patch is available, it takes time to properly test, QA (against all kinds of heterogeneous environments where consumers run our products) and then re-build dozens to hundreds of images and release a new version of our entire stack, either SUSE® Rancher Prime, Harvester or Longhorn.

Our KB 000020476 presents how we engage to address critical and high impact CVEs across our products and, if necessary, even releasing emergency versions of our products, like how we did in the past with Rancher v2.7.3, for example.

To know more about our workflow to triage and fix CVEs, please consult KB 000021574

Toward a Near-Zero CVE State

The road to a near-zero CVE state in our software is always our goal. It’s an endeavor that takes time, manual effort and a lot of automation, especially when we’re not talking about a single container image or a simple product, but hundreds of container images across a complex cloud native stack solution like SUSE® Rancher Prime.

Communication is another vital component on the path to achieving near-zero applicable CVEs. That’s why we ensure there’s constant communication, across multiple channels, about the real and applicable CVEs and, most importantly, clear information about false-positives and not-applicable security issues

SUSE® Rancher Prime has laid out the foundation of how we plan to achieve this goal, thus providing an even more secure software suite and solution stack to our consumers. Please stay tuned for new and updated information about this topic in SUSE® Rancher Prime.

If you believe that you have found a vulnerability in one of our products, please check SUSE’s security page on how to safely submit security issues to us. SUSE® Rancher Prime’s  specific security policy is available here.

For questions or concerns, please reach out to your designated SUSE® contact or open a Support Case by following these guidelines.

— The SUSE® Rancher Prime Security team

1 The software uses a dependency that has a known CVE, but the specific code path affected by the CVE isn’t used in the software, thus the software isn’t affected by the CVE itself.

2 The scans are done against the latest stable patch versions of each release line and development versions (aka dev or head branches). For example, as of September 2024 the latest patch version of Rancher Prime v2.9 is v2.9.2, so we scan only v2.9.2 and v2.9-head (dev branch). The same applies for v2.8 with v2.8.8 and v2.8-head, and so on. Automated issues are then created for teams to work on the identified CVEs (or viruses).

3 The United States’ National Telecommunications and Information Administration agency.