Release notes usually only list changes that happened between two subsequent releases. Certain important entries from the release notes documents of previous product versions may be repeated. To make such entries easier to identify, they contain a note to that effect.

2 Changes in 4.2.10 #

2.1 Cilium Update #

4.2.10 brings in a Cilium addon rebuild to include fixes for bsc#1215713 and bsc#1216123. More specific, the build of Cilium and Envoy now includes a patched version of nghttp2.

2.2 Required Actions #

Run skuba addons upgrade apply to update Cilium images to rev6 which has the bug fixes to be installed.

2.3 Bugs Fixed in 4.2.10 since 4.2.9 #

bsc#1215713 [cilium] VUL-0: CVE-2023-35945: nghttp2: HTTP/2 memory leak in nghttp2 codec
bsc#1216123 [cilium] VUL-0: CVE-2023-44487: nghttp2: HTTP/2 Rapid Reset Attack

3 Changes in 4.2.9 #

3.1 CRI-O Update #

4.2.9 brings in a CRI-O update, precisely to 1.19.7.

It includes a the fix for bsc#1200285

3.2 Required Actions #

In order to update to CRI-O 1.19.7, a regular ugprade process is required as decribed in Cluster Updates from the Admin Guide.

3.3 Bugs Fixed in 4.2.9 since 4.2.8 #

bsc#1200285 [cri-o] VUL-0: CVE-2022-1708: cri-o: memory exhaustion on the node when access to the kube API

4 Changes in 4.2.8 #

4.1 CRI-O Update #

4.2.7 brings in a CRI-O update, precisely to 1.19.7.

It includes a the fix for bsc#1196424

4.2 Required Actions #

In order to update to CRI-O 1.19.7, a regular ugprade process is required as decribed in Cluster Updates from the Admin Guide.

4.3 Bugs Fixed in 4.2.8 since 4.2.7 #

bsc#1196424 [cri-o] crio-1.19 seems to increase CPU throttling i

5 Changes in 4.2.7 #

5.1 CRI-O Update #

4.2.7 brings in a CRI-O update, precisely to 1.19.7.

It includes a the fix for bsc#1177952

5.2 Required Actions #

In order to update to CRI-O 1.19.7, a regular ugprade process is required as decribed in Cluster Updates from the Admin Guide.

5.3 Bugs Fixed in 4.2.7 since 4.2.6 #

bsc#1177952 [cri-o] reboot of worker hangs for a very long time before crio is up and running (>20 Minutes)

6 Changes in 4.2.6 #

6.1 Kubernetes Update #

4.2.6 brings in a Kubernetes update, precisely to 1.17.17.

It includes a backport to fix CVE-2021-25741.

6.2 Required Actions #

In order to update to Kubernetes 1.17.17, follow the instructions in the Admin Guide.

6.3 Bugs Fixed in 4.2.6 since 4.2.5 #

bsc#1189416 [kubernetes] CVE-2021-25741: Symlink Exchange Can Allow Host Filesystem Access

7 Changes in 4.2.5 #

7.1 Skuba Update #

4.2.5 Bring a new skuba release that fixes a cosmetic issue when upgrading to v4.5 CaaSP based in SLE-15-SP2.

7.2 Required Actions #

To apply the fix for the CRI-O bug mentioned below, you must perform skuba-update. See https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_updates.html#_base_os_updates for more details.

7.3 Bugs Fixed in 4.2.5 since 4.2.4 #

bsc#1179989 [skuba] Issue with skuba addon upgrade plan and kucero.
bsc#1174182 [cri-o] crio-shutdown.service ExecStop fails
bsc#1181702 Last public cloud update of terraform packages causes inconsistencies on admin node

8 Changes in 4.2.4 #

8.1 'helm3' Package Available #

Included in 4.2.4 is a package for Helm 3. This package will not be installed by default but is available for users who prefer to use Helm 3 over the default Helm 2 in the 4.2 releases. The documentation has been updated to include instructions on migrating to and using Helm 3.

8.2 Kubernetes Update #

4.2.4 brings in a Kubernetes update, precisely to 1.17.13. The list of features and bug fixes is long, see:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#v11713

8.3 kucero addon #

4.2.4 adds the kucero (*KU*bernetes control plane *CE*rtificate *RO*tation) addon. It rotates all the kubeadm managed certificates and kubeconfigs and signs kubelet server CSR. See link:https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_security.html#_automatic_certificate_renewal for further details.

8.4 Required Actions #

In order to apply addon updates you need to refresh you local addons folder. You must perform skuba addon refresh localconfig prior executing the skuba addon upgrade plan command. For further details, see https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_updates.html#_generating_an_overview_of_available_platform_updates

To apply the fix for the CRI-O bug mentioned below, you must perform skuba-update. See https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_updates.html#_base_os_updates for more details.

SUSE CaaS Platform did not detect existing metrics-server installed by the administrator via helm, so please remove the installed metrics-server manually before running skuba upgrade addons.
```
helm delete metrics-server --purge
```

8.5 Bugs Fixed in 4.2.4 since 4.2.3 #

bsc#1177661 VUL-0: CVE-2020-8565: kubernetes: Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel >= 9
bsc#1177662 VUL-0: CVE-2020-8566: kubernetes: Ceph RBD adminSecrets exposed in logs when loglevel >= 4
bsc#1174951 VUL-0: CVE-2020-15106,CVE-2020-15112: etcd: a large slice causes panic in decodeRecord method and improper checks in entry index
bsc#1176755 VUL-1: CVE-2020-15184: helm,helm2,helm3: alias field on a Chart.yaml is not properly sanitized
bsc#1176754 VUL-1: CVE-2020-15185: helm,helm2,helm3: Helm repository can contain duplicates of the same chart
bsc#1176753 VUL-1: CVE-2020-15186: helm,helm2,helm3: plugin names are not sanitized properly
bsc#1176752 VUL-0: CVE-2020-15187: helm,helm2,helm3: plugin can contain duplicates of the same entry
bsc#1177362 VUL-0: CVE-2020-8029: skuba: Insecure handling of private key
bsc#1177361 VUL-0: CVE-2020-8030: skuba: Insecure /tmp usage when joining node to cluster
bsc#1159274 VUL-0: CVE-2019-19794: coredns: The miekg Go DNS package improperly generates random numbers because math/rand is used
bsc#1176903 skuba does not renew the client-cert of the admin.conf in the skuba cluster directory
bsc#1165854 Sync terraform* with SUSE:SLE-15-SP1
bsc#1174219 Mixed /etc/sysconfig/kubelet changes between older/newer CaaSP4 nodes

8.6 Documentation Changes #

Added instructions for automatic certificate renewal using the kucero addon.
Various minor fixes and improvements, refer to: https://github.com/SUSE/doc-caasp/releases
Added migration instructions for new optional Helm 3 installation.

8.7 Known Issues #

Modifying the file /etc/sysconfig/kubelet directly is not supported: documentation at https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_miscellaneous.html#_configuring_kubelet
bsc#1179989 skuba addon upgrade plan shows an error on stdout when upgrading from v4.2.3 to v4.2.4. Cosmetic issue only, the upgrade procedure is still functional.

9 Changes in 4.2.3 #

9.1 Required Actions #

SUSE CaaS Platform did not detect existing metrics-server installed by the administrator via helm, so please remove the installed metrics-server manually before running skuba upgrade addons.
```
helm delete metrics-server --purge
```

9.2 Bugs Fixed in 4.2.3 since 4.2.2 #

bsc#1174400 [cri-o] $HOME is not used in /etc/passwd when runAsUser is used with crio 1.16.1

9.3 Documentation Changes #

Minor fixes and improvements, refer to: https://github.com/SUSE/doc-caasp/releases/

9.4 Known Issues #

Kubeproxy is not fully deprecated since envoyproxy requires support of Linux Kernel 5.3 and upwards.

9.4.1 sles15sp1 aws image is inactive #

When deploying on AWS, you need to modify the configuration for Terraform on your management node. You need to set state = "inactive" in /usr/share/caasp/terraform/aws/ami.tf, to avoid the the error Error: Invalid index during deployment of nodes.

10 Changes in 4.2.2 #

10.1 Required Actions #

Run skuba addons upgrade apply to update Cilium images to rev5 which has the bug fixes to be installed.
In order to use the latest skuba fixes, you need to update the admin workstation. For detailed instructions, see the Administration Guide
The kubernetes security fix (CVE-2020-8557) will be applied to the kubelet daemon running on the nodes by skuba-update. See https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_updates.html#_base_os_updates for more details.

10.2 Bugs Fixed in 4.2.2 since 4.2.0 #

bsc#1173039 [cilium] - cilium fails to start when IPv6 is disabled
bsc#1173055 [skuba] - dex to AD - error 401 after successful login
bsc#1146991 [skuba] - BPF filesystem is not mounted when cilium pods are restarted
bsc#1173165 [gangway] - Insecure SSL/TLS versions in Gangway (v4)
bsc#1173984 [kubernetes] - Node disk DOS by writing to container /etc/hosts CVE-2020-8557

10.3 Documentation Changes #

Various minor fixes and improvements, refer to: https://github.com/SUSE/doc-caasp/releases

10.4 Known Issues #

Kubeproxy is not fully deprecated since envoyproxy requires support of Linux Kernel 5.3 and upwards.

10.4.1 sles15sp1 aws image is inactive #

11 Changes in 4.2.1 #

11.1 Cilium Update to 1.6.6 with Envoy #

SUSE CaaS Platform 4.2.1 comes with an update of Cilium from version 1.5.3 to version 1.6.6. For a detailed list of all the changes in this most current version, see the Cilium Changelog.
The update also provides Envoy (open source edge and service proxy) support for Cilium.
Envoy and its dependencies are packaged in version 1.12.2.
In version 1.6.6 Cilium uses Custom Resource Definition (CRD) and ConfigMap points on etcd have been removed.

11.2 `skuba` #

Cilium 1.6.6 deployment with CRD

11.3 Metrics Server #

SUSE CaaS Platform now supports the metrics server as an addon to monitor the CPU and memory of a pod or node. You can use commands kubectl top nodes get status for node and use kubectl top pods to get status for one or more pods. Also, with metrics-server default built-in by SUSE CaaS Platform, the user can set up a horizontal pod autoscaler (HPA) to auto-scale the number of pods or set up a vertical pod autoscaler (VPA) to auto allocate more or less CPUs and memory resources.

11.4 Cert Status Checker #

The Cert Status Checker is a new feature, which exposes a cluster-wide certificates status. It uses the monitoring stack (Prometheus and Grafana) to receive alerts by Prometheus Alert manager and monitor certificate status on the Grafana dashboard.

For detailed instructions please see the Administration Guide

11.5 VSphere VCP #

Allow Kubernetes pods to use VMware vSphere Virtual Machine Disk (VMDK) volumes as persistent storage.

For detailed instructions please see the Administration Guide

11.6 Required Actions #

To migrate from etcd to CRD in case of upgrade, follow the steps described in the official Cilium documentation.
Run skuba addons upgrade apply to upgrade Cilium to version 1.6.6. See information about a possible warning, which might appear during the upgrade in the Administration Guide.
SUSE CaaS Platform did not detect existing metrics-server installed by the administrator, so please remove the installed metrics-server manually before upgrade addons.
```
helm delete metrics-server --purge
```
In order to update skuba to apply the latest fixes, you also need to update the admin workstation. For detailed instructions, see this section in the Admin Guide.

11.7 Bugs Fixed in 4.2.1 since 4.2.0 #

bsc#1162651 [skuba] - 10 revisionHistoryLimit value seems too high for our workloads
bsc#1169506 [skuba] - fix issue for aws iam profile
bsc#1121353 [skuba] - Restrict kube-system control plane pods

11.8 Documentation Changes #

Added instructions how to enable and configure Kubernetes Audit Log.
Updated Network Policy administration with Cilium at Network Policies.
Added instructions for deployment of Cilium Network policies with Envoy at Cilium Network Policy Config Examples.
Added Networking Whitelist to deployment guide requirements.
Restructured the Disaster Recovery chapter.
Added instruction for vSphere Cloud Provider Integration.
Fixed the location for the flexvolume driver in the documentation.
Various other fixes and improvements, refer to: https://github.com/SUSE/doc-caasp/releases/tag/release-4.2.1

11.9 Known Issues #

Kubeproxy is not fully deprecated since envoyproxy requires support of Linux Kernel 5.3 and upwards.
helm install is failing on "unavailable metrics.k8s.io/v1beta1 API": Before operating with the cluster and using helm, make sure all pods and apiservices are available. bsc#1172398

12 Changes in 4.2.0 #

12.1 Deprecations in 4.2.0 #

The hyperkube image, combining multiple Kubernetes binaries, is planned for removal in 4.3.0, due to upstream deprecations. If running SUSE CaaS Platform in an airgapped environment, please ensure that all our images are mirrored.
Remove ability to re-enable serving deprecated APIs types:
- extensions/v1beta1
- apps/v1beta1
- apps/v1beta2

For more information check: https://github.com/kubernetes/kubernetes/issues/43214

12.2 Kubernetes Update #

4.2.0 brings in a Kubernetes update, precisely to 1.17.4. The list of features and bug fixes is long, see:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#v1174

12.3 Addon Customization Persistence #

This release introduces kustomize to persist addon configuration changes across updates and reboots.

12.4 Skuba: Disabling `firewalld` #

Skuba should disable firewalld when bootstrapping/joining a node, so it adds a startup step to check whether firewalld is disabled. This was done using cloud init, which however does not work on bare metal deployments. So in order to ensure that firewalld is disabled, this check was introduced into skuba.

12.5 Datastore for VMware #

VMware Terraform config now supports setting a datastore cluster as the storage backend. Please refer to the Deployment Instructions for more information.

12.6 Required Actions #

12.6.1 Kubernetes 1.17 #

In order to update to Kubernetes 1.17, follow the instructions in the Admin Guide.

If your cluster is not on the latest Kubernetes version prior to applying the update, you will encounter an issue when skuba-update tries to update your nodes. See the Section 12.9, “Known Issues” section for instructions on how to proceed.

12.6.2 Conmon and CRI-O #

Conmon and CRI-O will be updated by skuba-update. No action is required from your side. For more info see the Cluster Updates section in the Admin Guide.

12.6.3 Skuba #

In order to update skuba, you also need to update the admin workstation. For detailed instructions, see this section in the Admin Guide.

12.6.4 Generate the `kustomize` Style Addon Configurations #

You must convert your addon manifests to the new kustomize aware file structure and formats. To do so please run the following commands from your management workstation.

Replace MASTER-NODE-IP with an IP address/FQDN of one of your master nodes. Replace CLUSTER-DEFINITION-PATH with the path to your existing cluster defintion files that were generated during the intial bootstrap/deployment.

skuba cluster init --control-plane MASTER-NODE-IP /tmp/new-cluster-init
mv CLUSTER-DEFINITION-PATH/addons CLUSTER-DEFINITION-PATH/addons-old
cp -r /tmp/new-cluster-init/addons CLUSTER-DEFINITION-PATH/

This will generate the existing addon configurations in the new format so you can amend them.

12.7 Bugs Fixed in 4.2.0 since 4.1.2 #

bsc#1161056 [cri-o] - Fix upgrade from 4.0.3 to 4.1.0 - skuba node upgrade - fails due to crio-wipe.service not starting
bsc#1159108 [admin-guide] grafana helm chart version newer than upstream but older image version / grafana version!
bsc#1157337 [skuba] After cluster creation all DEX and all GANGWAY pods run on the first master
bsc#1152334 [skuba] skuba update management - HAS-UPDATES HAS-DISRUPTIVE-UPDATES → no vs none
bsc#1160460 [podman] Update podman to 1.8.0
bsc#1164390 [conmon] Add conmon to SLE15 Containers Module
bsc#1162093 [kubernetes] kubelet referenced wrong volume-plugin dir after upgrade
bsc#1121353 [kubernetes] Kubernetes – Master node pod configured with Privileged PSP

12.8 Documentation Changes #

The QuickStart Guide has been removed pending review and rewrite. Please use the Deployment Guide.
Disaster Recovery with Velero is now documented in the Admin Guide.
A subchapter on Managing Replicas has been added to Deployment Requirements.
The list of required addon images was updated.
SUSE Cloud Application Platform integration was removed from the SUSE CaaS Platform Admin Guide. Please now refer to: Deploying SUSE Cloud Application Platform on SUSE CaaS Platform.
A note about using the --non-interactive-include-reboot-patches was added to the Admin Guide.
Instructions on how to update Dex have been enhanced. For details, see the Admin Guide.
We updated the air gapped deployment with a new diagram. See the Admin Guide.
We added an example on how to set up Prometheus Recording Rules.
Instructions on how to troubleshoot the "cannot attach profile" error from AWS have been added.
The Glossary was reintroduced to all our guides.
Various other fixes and improvements (Refer to: https://github.com/SUSE/doc-caasp/releases).

12.9 Known Issues #

12.9.1 `skuba-update` Error: `patterns-caasp-Node` Conflicts with CRI-O Update #

If your cluster is not up-to-date, meaning it is not in the latest Kubernetes version, skuba-update will try to install the latest version of CRI-O, which will create a conflict with the currently installed Kubernetes packages.

More precisely, you might encounter an error similar to this:

patterns-caasp-Node conflicts with CRI-O

In that case, the recommended solution is to upgrade the cluster to the latest Kubernetes version available, this can be done by running the regular SUSE CaaS Platform Kubernetes upgrade procedure based on the command skuba node upgrade, which is described in the Admin Guide.

13 Changes in 4.1.2 #

13.1 Deployment on AWS as Technology Preview #

Deployment of SUSE CaaS Platform on Amazon Web Services (AWS) has been tested and documented. Terraform is used to deploy the infrastructure and the skuba tool to bootstrap the Kubernetes cluster on top of it. For detailed instructions please see the Deployment Guide. Please note that SUSE CaaS Platform deployment on AWS may not be functionally complete, and is not intended for production use.

13.2 Terraform Upgrade #

SUSE CaaS Platform can now be deployed with Terraform 0.12. All details of the new version can be found in the HashiCorp Documentation. The official website for the Terraform 0.12 upgrade is https://www.terraform.io/upgrade-guides/0-12.html.

13.3 etcd Backup and Restore for Master Nodes Disaster Recovery #

Provide etcd backup process on-demand or on a schedule to prevent etcd data corruption.
Provide etcd restore process to recover failed master node(s) to restore etcd quorum for cluster serving.

For detailed instructions please see link: the Administration Guide.

13.4 Velero for Disaster Recovery #

Provide Velero as a solution for data protection and data migration by backing up and migrating Kubernetes resources and persistent volumes to and from externally supported storage backend on demand or on a schedule.

For detailed instructions please see link: the Administration Guide.

13.5 Required Actions #

13.5.1 Upgrade Terraform Files and State #

In order to seamlessly switch to Terraform 0.12 you need to make sure that:

All files follow the new syntax for the HashiCorp Configuration Language included in Terraform 0.12
All boolean values are true or false and not 0 or 1
All variables are explicitly declared
All dependencies are explicitly declared to reach the expected behavior

13.5.2 Recommended Procedure #

Enter your Terraform files/state folder and:

Install the latest version of Terraform using zypper in terraform (the installed version should be 0.12.19)
Navigate to your Terraform root folder (e.g. /usr/share/caasp/terraform/vmware)
Migrate Terraform files with the automatic migration tool by running terraform 0.12upgrade
- For OpenStack, run the Section 13.5.3, “Extra Operations for In-place Upgrade of OpenStack Terraform Files” (see below)
- Run terraform apply to update the Terraform definitions to the new format used by 0.12
  Important
  If you do not update the definitions before running Terraform again your output might contain nil/null strings when you run terraform refresh followed by terraform output. This can break automations that are based on the output. Please make sure you have updated/applied all definitions before running Terraform.
Run zypper up skuba
You can then run the terraform init/plan/apply commands as usual.

13.5.3 Extra Operations for In-place Upgrade of OpenStack Terraform Files #

Replace any boolean values written as a number with false/true. For example, for the variables in openstack/variables.tf (and their equivalent in your terraform.tfvars file), replace default = 0 with default = false in the variables workers_vol_enabled and dnsentry. Do the same for any extra boolean variable you might have added.
Introduce a depends_on on the resource "openstack_compute_floatingip_associate_v2" "master_ext_ip" in master-instance.tf:
```
depends_on = [openstack_compute_instance_v2.master]
```

Introduce a depends_on on the resource "master_wait_cloudinit" in master-instance.tf:

depends_on = [
  openstack_compute_instance_v2.master,
  openstack_compute_floatingip_associate_v2.master_ext_ip
]

Introduce a depends_on on the resources "openstack_compute_floatingip_associate_v2" "worker_ext_ip" and "null_resource" "worker_wait_cloudinit" in worker-instance.tf, similarly to the ones for master. Replace master with worker in the examples above.
Update the resources resource "openstack_compute_instance_v2" "master" and resource "openstack_compute_instance_v2" "worker" with master-instance.tf and worker-instance.tf respectively. Add the following resources:
```
lifecycle {
  ignore_changes = [user_data]
}
```
Note
The above option is needed because Terraform will detect all machines as new resources when user_data changes during the upgrade.
This will make it possible to update your cluster from a Terraform 0.11 state into a Terraform 0.12 state without tearing it down completely.

Warning

When adding lifecycle { ignore_change = [user_data] } in your master and worker instances, you will effectively prevent updates of nodes, should you or SUSE update the user_data. This should be removed as soon as possible after the migration to Terraform 0.12.

13.5.4 etcdctl #

Run zypper in etcdctl in the management host to install etcdctl.

13.5.5 Update packages for general fixes #

Update skuba package and patterns-caasp-Management on your management workstation as you would do with any other package.

Refer to: https://documentation.suse.com/sles/15-SP1/single-html/SLES-admin/#sec-zypper-softup-update

Updating patterns-caasp-Management will install the new terraform providers for AWS.

Packages on your cluster nodes (cri-o) will be updated automatically by skuba-update link:https://documentation.suse.com/suse-caasp/4.1/html/caasp-admin/_cluster_updates.html#_base_os_updates

13.6 Bugs Fixed in 4.1.2 since 4.1.1 #

bsc#1161056 [cri-o] - Fix upgrade from 4.0.3 to 4.1.0 - skuba node upgrade - fails due to crio-wipe.service not starting
bsc#1161179 [cri-o] - Fix invalid apparmor profile
bsc#1158440 [terraform] - Update in SLE-15 (bsc#1158440, CVE-2019-19316)
bsc#1148092 [terraform] - Include in SLE-15 (bsc#1148092, jsc#ECO-134)
bsc#1145003 [terraform-provider-openstack] - Update to version 1.19.0
bsc#1159082 [grafana] - Fix some missing container images of grafana helm chart
bsc#1161225 [grafana] - Fix grafana helm chart has app version 6.4.2 but version is 6.2.5
bsc#1161110 [grafana] - Fix Grafana dashboard should not name "CaaSP" but "SUSE (r) CaaS Platform"
bsc#1162093 [kubelet] - Release fix for volume-plugin-dir in kubernetes packages
bsc#1160463 [skuba] - Fix skuba-update --version always 0.0.0
bsc#1157323 [skuba] - Fix need a way to report on current available CaaSP version vs. installed version

13.7 Documentation Changes #

Added AWS deployment instructions (Tech Preview)
Added KVM deployment instructions
Improved instructions for Monitoring to deploy Grafana in a sub path and enhanced ingress settings
Fix unspecific expression in AlertManager example
Added notes on certificate rotation for the control plane
Various other fixes and improvements (Refer to: https://github.com/SUSE/doc-caasp/releases)

14 Changes in 4.1.1 #

skuba fixes (see below)
supportutils-plugin-suse-caasp fixes (see below)
kubernetes and cri-o fixes (see below)
caasp-release-notes fixes (see below)
prometheus fixes (see below)
CRI-O now uses the system proxy settings (see Section 14.3, “Documentation Changes”)

14.1 Required Actions #

14.1.1 Update packages for general fixes and added supportconfig plugin #

Update skuba and kubernetes-client packages on your management workstation as you would do with any other package.

Refer to: https://documentation.suse.com/sles/15-SP1/single-html/SLES-admin/#sec-zypper-softup-update

Packages on your cluster nodes (cri-o, kubernetes, supportutils-plugin-suse-caasp) will be updated automatically by skuba-update link:https://documentation.suse.com/suse-caasp/4.1/html/caasp-admin/_cluster_updates.html#_base_os_updates

14.1.2 Fix Prometheus `kube-state-metrics` #

Use helm upgrade command to fix the Prometheus kube-state-metrics image.

Finally, in order to use new Prometheus pushgateway image, enable the service in your prometheus-config-values.yaml config file:

pushgateway:
  enabled: true

Then run the helm upgrade command https://helm.sh/docs/intro/using_helm/#helm-upgrade-and-helm-rollback-upgrading-a-release-and-recovering-on-failure.

Afterwards you can deploy Prometheus as usual. Refer to: https://documentation.suse.com/suse-caasp/4.1/html/caasp-admin/_monitoring.html#_prometheus.

14.2 Bugs Fixed in 4.1.1 since 4.1.0 #

bsc#1161179 [cri-o] - cilium crashes with "apparmor failed to apply profile: write /proc/self/attr/exec: no such file or directory
bsc#1161056 [cri-o] - upgrade from 4.0.3 to 4.1.0 - skuba node upgrade - fails due to crio-wipe.service not starting
bsc#1155323 [cri-o] - Include system proxy settings in service if present
bsc#1159452 [skuba] - Fixed do not panic when version is unknown
bsc#1157802 [skuba] - Enhanced skuba auth login help/error message
bsc#1155810 [skuba] - Refactored to fix CaaSP SSL / PKI / CA Infrastructure unclear and probably inconsistent and wrong?
bsc#1157802 [skuba] - skuba auth login help should mention the port that needs to be use (:32000)
bsc#1137337 [skuba] - Skuba log level description is missing
bsc#1155593 [kubernetes] - second master join always fails
bsc#1160443 [supportutils-plugin-suse-caasp] - Extend supportconfig to check certificates expiration time
bsc#1152335 [supportutils-plugin-suse-caasp] - Add etcd logs for v4
bsc#1160600 [caasp-release-notes] - caasp-release package points to caasp-release-notes 4.0
bsc#1159074 [prometheus] - Prometheus pushgateway image v0.8.0 missing on registry.suse.com/caasp/v4
bsc#1161975 [prometheus] - kube-state-metrics - endless "Failed to list *v1beta1.ReplicaSet: the server could not find the requested resource" on 1.16.2

14.3 Documentation Changes #

Added instructions for Stratos Web Console (Tech Preview)
Added instructions for etcd storage performance testing
Added instructions for etcd troubleshooting
Updated CRI-O proxy configuration instructions
Updated upgrade instructions with more information about manual upgrades and reboots
Various minor fixes and improvements (Refer to: https://github.com/SUSE/doc-caasp/releases)

15 Changes in 4.1.0 #

15.1 Kubernetes update #

SUSE CaaS Platform now ships with Kubernetes 1.17.17. Most of the significant changes relate to this upgrade, as more than 31 enhancements were merged in the Kubernetes 1.17.17 release. You can read a short summary of the changes under Section 19.1.7, “Changes to the Kubernetes Stack”. Manual actions are required for 4.1.0 release.

15.2 Helm security update #

Moreover, helm has been updated to fix a security issue (CVE-2019-18658).

15.3 Stratos, a web console for Kubernetes #

Stratos is now available as tech preview for SUSE CaaS Platform. Stratos is a web console for Kubernetes and for Cloud Foundry. A single instance of Stratos can be used to monitor and interact with different Kubernetes clusters as long as their API endpoints are reachable by Stratos.

Stratos integrates with Prometheus: it can scrape metrics collected by Prometheus and show them using pre-built charts.

Finally Stratos can be used to interact with helm chart repositories. It can show the charts available and install them straight from its web interface. It can also show all the workloads that are running on a Kubernetes that have been created by helm chart.

Note

The helm chart integration is a tech preview feature of Stratos that must be enabled at deployment time.

15.4 Required Actions #

15.4.1 Skuba and helm update Instructions #

Update skuba and helm on your management workstation as you would do with any other package.

Refer to: https://documentation.suse.com/sles/15-SP1/single-html/SLES-admin/#sec-zypper-softup-update

Warning

When running helm-init you may hit a known bug on the certificate validation:

https://kubernetes-charts.storage.googleapis.com is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: x509: certificate signed by unknown authority

In order to fix this, run:

sudo update-ca-certificates

After updating helm to latest version on the management host, you have to also upgrade the helm-tiller image in the cluster, by running:

helm init \
    --tiller-image registry.suse.com/caasp/v4/helm-tiller:2.16.1 \
    --service-account tiller --upgrade

15.4.2 Upgrade Your Kubernetes Cluster #

Use skuba to upgrade your Kubernetes cluster as documented in the Administration guide.

Warning

Please, do not run zypper patch manually on your nodes. If you do, you will see an error about a conflict when patching CRI-O. This is expected, because the patch is not supposed to be installed this way.

Instead, cluster updates are being handled by skuba as documented in the Administration guide.

15.4.3 Update Your Kubernetes Manifests for Kubernetes 1.17.17: #

Some API resources are moved to stable, while others have been moved to different groups or deprecated.

The following will impact your deployment manifests:

DaemonSet, Deployment, StatefulSet, and ReplicaSet in extensions/ (both v1beta1 and v1beta2) is deprecated. Migrate to apps/v1 group instead for all those objects. Please note that kubectl convert can help you migrate all the necessary fields.
PodSecurityPolicy in extensions/v1beta1 is deprecated. Migrate to policy/v1beta1 group for PodSecurityPolicy. Please note that kubectl convert can help you migrate all the necessary fields.
NetworkPolicy in extensions/v1beta1 is deprecated. Migrate to networking.k8s.io/v1 group for NetworkPolicy. Please note that kubectl convert can help you migrate all the necessary fields.
Ingress in extensions/v1beta1 is being phased out. Migrate to networking.k8s.io/v1beta1 as soon as possible. This new API does not need to update other API fields and therefore only a path change is necessary.
Custom resource definitions have moved from apiextensions.k8s.io/v1beta1 to apiextensions.k8s.io/v1.

Please also see https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/ for more details.

15.5 Bugs Fixed in 4.1.0 since 4.0.3 #

bsc#1144065 [cri-o] - (CVE-2019-10214) VUL-0: CVE-2019-10214: libcontainers-common: library does not enforce TLS connections
bsc#1118898 [cri-o] - (CVE-2018-16874) VUL-0: CVE-2018-16874: go: cmd/go: directory traversal
bsc#1100838 [cri-o] - cri-o does not block /proc/acpi pathnames (i.e., also affected by (CVE-2018-10892))
bsc#1118897 [etcd] - (CVE-2018-16873) VUL-0: CVE-2018-16873: go: cmd/go: remote command execution
bsc#1118899 [etcd] - (CVE-2018-16875) VUL-0: CVE-2018-16875: go: crypto/x509: CPU denial of service
bsc#1156646 [helm] - (CVE-2019-18658) VUL-0: CVE-2019-18658: helm: commands that deal with loading a chart as a directory or packaging a chart provide an opportunity for a maliciously designed chart to include sensitive content such as /etc/passwd
bsc#1152861 [kubernetes] - (CVE-2019-11253) VUL-0: CVE-2019-11253: kubernetes: YAML parsing vulnerable to "Billion Laughs" attack, allowing for remote denial of service
bsc#1146991 [kubernetes] - BPF filesystem is not mounted, possible downtime when cilium pods are restarted
bsc#1147142 [kubernetes] - Update golang/x/net dependency to bring in fixes for (CVE-2019-9512), (CVE-2019-9514)
bsc#1143813 [kubernetes] - kubelet sometimes starting too fast
bsc#1143813 [skuba] - CaaSP SSL / PKI / CA Infrastructure unclear and probably inconsistent and wrong?
bsc#1152335 [supportutils-plugin-suse-caasp] - supportconfig adjustments for CaaSP v4 missing

15.6 Documentation Updates #

Switched examples to use SUSE supported helm, Prometheus, nginx-ingress and Grafana charts and images
Added instructions on how to replace Kubernetes certificates with custom CA certificate
Added instructions to configure custom certificates for gangway and dex
Added instructions for secured Tiller deployment
Added notes about unique machine-id requirement
Added timezone configuration example for AutoYaST
Various minor bugfixes and improvements

15.7 Known Issues #

15.7.1 Skuba-upgrade could not parse "Unknown" as version #

Running "skuba node upgrade plan" might fail with the error "could not parse "Unknown" as version" when a worker, after running "skuba node upgrade apply", had not fully started yet.

If you are running into this issue, please add some delay after running "skuba node upgrade apply" and prior to running "skuba node upgrade plan".

This is tracked in bsc#1159452

16 Changes in 4.0.3 #

Prometheus and Grafana: official monitoring solution for SUSE CaaS Platform
Airgap: format change of https://documentation.suse.com/external-tree/en-us/suse-caasp/4/skuba-cluster-images.txt
389-ds fixes (see below)
skuba fixes (see below)

16.1 Prometheus and Grafana: official monitoring solution for SUSE CaaS Platform #

Prometheus and Grafana were already documented but based on upstream helm charts and containers.

In version 4.2.7, official SUSE helm carts and containers are now available in the helm chart repository (kubernetes-charts.suse.com) and the container registry (registry.suse.com).

16.2 Airgap: Format Change #

The format of https://documentation.suse.com/external-tree/en-us/suse-caasp/4/skuba-cluster-images.txt was changed to be able to express more data. Specifically to add skuba and SUSE CaaS Platform versions, so that one can match the images that should be pulled with the respective version.

This way, you can run air gapped production and staging clusters with different SUSE CaaS Platform versions.

16.3 Required Actions #

16.3.1 Skuba Update Instructions #

Update skuba on your management workstation as you would do with any other package.

Refer to: SUSE Linux Enterprise Server 15 SP1 Admin Guide: Updating Software with Zypper

16.3.2 Prometheus and Grafana Installation Instructions #

You will need to use helm and kubectl to deploy Prometheus and Grafana. Refer to: Monitoring chapter in the SUSE CaaS Platform admin guide

16.3.3 389-ds Update Instructions #

389-ds containers have been updated in registry.suse.com (see Bugs fixed below). In order to deploy your 389-ds container, see Configuring and external ldap server at the SUSE CaaS Platform admin guide

16.4 Documentation Changes #

Updated monitoring documentation in the admin guide to reflect official charts/containers for monitoring stack
Added/Updated information about 389-ds deployment and configuration
Added information about subnet sizing to deployment guide system requirements
Added information on using a cluster wide root CA to admin guide
Add note about NTP client requirement for management workstation
Added less aggressive nginx timeout values to examples
Unified use of placeholders in code examples to <PLACEHOLDER> format
Various minor formatting and wording fixes

16.5 Bugs Fixed in 4.0.3 since 4.0.2 #

bsc#1156667 [Prometheus and Grafana] - User "system:serviceaccount:monitoring:prometheus-kube-state-metrics" cannot list resource
bsc#1140533 [Prometheus and Grafana] - Prometheus and grafana images and helm charts on registry.suse.com
bsc#1155173 [skuba] - skuba node upgrade does not really upgrade node successfully
bsc#1151689 [skuba] - Default verbosity hides most errors
bsc#1151340 [389-ds] - ERR - add_new_slapd_process - Unable to start slapd because it is already running as process 8
bsc#1151343 [389-ds] - The config /etc/dirsrv/slapd-*/dse.ldif can not be accessed. Attempting restore
bsc#1151414 [389-ds] - NOTICE - dblayer_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
bsc#1157332 [patterns-caasp] - caasp-release rpm not installed - probably should be included in the patterns?

17 Changes in 4.0.2 #

Note

Core addons are addons deployed automatically by skuba when you bootstrap a cluster. Namely:

Cilium
Dex
Gangway
Kured
Default Pod Security Policies (PSP’s)

skuba addon command has been introduced to handle core addons
- skuba addon upgrade plan will inform about what core addons will be upgraded
- skuba addon upgrade apply will upgrade core addons in the current cluster

17.1 Required Actions #

When using skuba addon upgrade apply, all settings of all addons will be reverted to the defaults. Make sure to reapply your changes after running skuba addon upgrade apply, had you modified the default settings of core addons.

17.2 Bugs fixed in 4.0.2 since 4.0.1 #

bsc#1145568 [remove-node] failed disarming kubelet due to 63 character limitation
bsc#1145907 LB dies when removing a master node in VMWare
bsc#1146774 AWS: pod to service connectivity broken in certain cases
bsc#1148090 Multinode cluster upgrade fails on 2nd master due to TLS handshake timeout
bsc#1148412 Gangway uses CSS stylesheet from cloudflare.com
bsc#1148524 Allow easy recovery from bootstrap failed during add-ons deployment phase
bsc#1148700 worker node upgrade needs to use kubeletVersion in nodeVersionInfoUpdate type
bsc#1149637 Misspelling of bootstrapping in a common error message
bsc#1153913 Can not bootstrap an new cluster if a valid kubectl config is present
bsc#1153928 Reboot can be triggered before skuba-update finishes
bsc#1154085 skuba node upgrade shows component downgrade
bsc#1154754 oauth2: cannot fetch token after 24 hours

18 Changes in 4.0.1 #

Updated Gangway container image (see Section 18.1, “Required Actions”)
Added air gap deployment instructions
Various bug fixes and improvements

18.1 Required Actions #

18.1.1 Update the Gangway Image #

The gangway image that shipped with SUSE CaaS Platform 4.0 must be updated manually by performing the following step:

kubectl set image deployment/oidc-gangway oidc-gangway=registry.suse.com/caasp/v4/gangway:3.1.0-rev4 --namespace kube-system

18.2 Known Issues #

You must update the gangway container image manually after update (see Section 18.1, “Required Actions” ).

For a full list of Known Issues refer to: Bugzilla.

18.3 Supported Platforms #

This release supports deployment on:

SUSE OpenStack Cloud 8
VMWare ESXi 6.7
KVM
Bare metal
(SUSE CaaS Platform 4.2.7 supports hardware that is certified for SLES through the YES certification program. You will find a database of certified hardware at https://www.suse.com/yessearch/.)

19 Changes in 4.0.0 #

19.1 What Is New #

19.1.1 Base Operating System Is Now SLES 15 SP1 #

The previous version used a minimal OS image called MicroOS. SUSE CaaS Platform 4 uses standard SLES 15 SP1 as the base platform OS. SUSE CaaS Platform can be installed as an extension on top of that. Because SLES 15 is designed to address both cloud-native and legacy workloads, these changes make it easier for customers who want to modernize their infrastructure by moving existing workloads to a Kubernetes framework.

Transactional updates are available in SLES 15 SP1 as a technical preview but SUSE CaaS Platform 4 will initially ship without the transactional-update mechanism enabled. The regular zypper workflow allows use of interruption-free node reboot. The SLES update process should help customers integrate a Kubernetes platform into their existing operational infrastructure more easily, nevertheless transactional updates are still the preferred process for some customers, which is why we provide both options.

19.1.2 Software Now Shipped as Packages Instead of Disk Image #

In the previous version, the deployment of the software was done by downloading and installing a disk image with a pre-baked version of the product. In SUSE CaaS Platform 4, the software is distributed as RPM packages from an extension module in SLES 15 SP1. This adaptation towards containers and SUSE Linux Enterprise Server mainly gives customers more deployment flexibility.

19.1.3 More Containerized Components #

We moved more of the components into containers, namely all the control plane components: etcd, kube-apiserver, kube-controller-manager, and kube-scheduler. The only pieces that are now running uncontainerized are CRI-O, kubelet and kubeadm.

19.1.4 New Deployment Methods #

We are using a combination of skuba (custom wrapper around kubeadm) and HashiCorp Terraform to deploy SUSE CaaS Platform machines and clusters. We provide Terraform state examples that you can modify to roll out clusters.

Deployment on bare metal using AutoYaST has now also been tested and documented: https://documentation.suse.com/suse-caasp/4/single-html/caasp-deployment/#deployment_bare_metal

Note

You must deploy a load balancer manually. This is currently not possible using Terraform. Find example load balancer configurations based on SUSE Linux Enterprise 15 SP1 and Nginx or HAProxy in the SUSE CaaS Platform Deployment Guide: https://documentation.suse.com/suse-caasp/4/single-html/caasp-deployment/#_load_balancer

19.1.5 Updates Using Kured #

Updates are implemented with skuba-update, that makes use of the kured tool and the SLE package manager. This is implemented in the skuba-update tool which glues zypper and the kured tool (https://github.com/weaveworks/kured). Kured (KUbernetes REboot Daemon) is a Kubernetes daemonset that performs safe automatic node reboots when the need to do so is indicated by the package management system of the underlying OS. Automatic updates can be manually disabled and configured: https://documentation.suse.com/suse-caasp/4/single-html/caasp-admin/#_cluster_updates

19.1.6 Automatic Installation of Packages For Storage Backends Discontinued #

In previous versions SUSE CaaS Platform would ship with packages to support all available storage backends. This negated the minimal install size approach and is discontinued. If you require a specific software package for your storage backend please install it using AutoYaST, Terraform or zypper. Refer to: https://documentation.suse.com/suse-caasp/4/single-html/caasp-admin/#_software_management

19.1.7 Changes to the Kubernetes Stack #

19.1.7.1 Updated Kubernetes #

SUSE CaaS Platform 4.2.7 ships with Kubernetes 1.17.17.

Kubernetes version 1.16 contains the following notable changes:

Custom Resources Definitions (CRD) are out of the beta version and are generally available in the apiextensions.k8s.io/v1 group.
IPv4/IPv6 dual stack is officially in alpha. Read up about the details of the new features of Kubernetes 1.16 here: https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/.

Kubernetes version 1.15 mainly contains enhancements to core Kubernetes APIs:

CustomResourceDefinitions Pruning, -Defaulting and -OpenAPI Publishing.
cluster life cycle stability and usability has been enhanced (kubeadm init and kubeadm join can now be used to configure and deploy an HA control plane)
new functionality of the Container Storage Interface (volume cloning) is available. Read up about the details of the new features of Kubernetes 1.15 here: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#115-whats-new

19.1.7.2 CRI-O Replaces Docker #

SUSE CaaS Platform now uses CRI-O 1.19.7 as the default container runtime. CRI-O is a container runtime interface based on the OCI standard technology. The choice of CRI-O allows us to pursue our open-source agenda better than competing technologies.

CRI-O’s simplified architecture is tailored explicitly for Kubernetes and has a reduced footprint but also guarantees full compatibility with existing customer images thanks to its adherence to OCI standards. Other than Docker, CRI-O allows to update the container runtime without stopping workloads; providing improved flexibility and maintainabilitty to all SUSE CaaS Platform users.

We will strive to maintain SUSE CaaS Platform’s compatibility with the Docker Engine in the future.

19.1.7.3 Cilium Replaces Flannel #

SUSE CaaS Platform now uses Cilium 1.6.6 as the Container Networking Interface enabling networking policy support.

19.1.7.4 Centralized Logging #

The deployment of a Centralized Logging node is now supported for the purpose of aggregating logs from all the nodes in the Kubernetes cluster. Centralized Logging forwards system and Kubernetes cluster logs to a specified external logging service, specifically the Rsyslog server, using Kubernetes Metadata Module - mmkubernetes.

19.1.8 Obsolete Components #

19.1.8.1 Salt #

Orchestration of the cluster no longer relies on Salt. Orchestration is instead achieved with kubeadm and skuba.

19.1.8.2 Admin Node / Velum #

The admin node is no longer necessary. The cluster will now be controlled by the master nodes and through API with skuba on any SUSE Linux Enterprise system, such as a local workstation. This also means the Velum dashboard is no longer available.

19.2 Known Issues #

19.2.1 Updating to SUSE CaaS Platform 4 #

In-place upgrades from earlier versions or from Beta 4 version to the generally available release is not supported. We recommend standing up a new cluster and redeploying workloads. For customers with production servers that cannot be redeployed, contact SUSE Consulting Services or your account team for further information.

19.2.2 Parallel Deployment #

To avoid fails, avoid parallel deployment of nodes. Joining master or worker nodes to an existing cluster should be done serially, meaning the nodes have to be added separately one after another. This issue will be fixed in the next release.

20 Support and Life Cycle #

SUSE CaaS Platform is backed by award-winning support from SUSE, an established technology leader with a proven history of delivering enterprise-quality support services.

SUSE CaaS Platform 4 has a two-year life cycle. Each version will receive updates while it is current, and will be subject to critical updates for the remainder of its life cycle.

For more information, check our Support Policy page https://www.suse.com/support/policy.html.

21 Support Statement for SUSE CaaS Platform #

To receive support, you need an appropriate subscription with SUSE. For more information, see https://www.suse.com/support/programs/subscriptions/?id=SUSE_CaaS_Platform.

The following definitions apply:

L1: Problem determination, which means technical support designed to provide compatibility information, usage support, ongoing maintenance, information gathering and basic troubleshooting using available documentation.
L2: Problem isolation, which means technical support designed to analyze data, reproduce customer problems, isolate problem area and provide a resolution for problems not resolved by Level 1 or prepare for Level 3.
L3: Problem resolution, which means technical support designed to resolve problems by engaging engineering to resolve product defects which have been identified by Level 2 Support.

For contracted customers and partners, SUSE CaaS Platform 4 is delivered with L3 support for all packages, except for the following:

Technology Previews
Packages that require an additional customer contract
Packages with names ending in -devel (containing header files and similar developer resources) will only be supported together with their main packages.

SUSE will only support the usage of original packages. That is, packages that are unchanged and not recompiled.

22 Documentation and Other Information #

22.1 Available on the Product Media #

Get the detailed change log information about a particular package from the RPM (where FILENAME.rpm is the name of the RPM):

rpm --changelog -qp FILENAME.rpm

22.2 Externally Provided Documentation #

For the most up-to-date version of the documentation for SUSE CaaS Platform 4, see https://documentation.suse.com/#suse-caasp

Find a collection of resources in the SUSE CaaS Platform Resource Library: https://www.suse.com/products/caas-platform/#resources

23 Obtaining Source Code #

This SUSE product includes materials licensed to SUSE under the GNU General Public License (GPL). The GPL requires SUSE to provide the source code that corresponds to the GPL-licensed material. The source code is available for download at http://www.suse.com/download-linux/source-code.html. Also, for up to three years after distribution of the SUSE product, upon request, SUSE will mail a copy of the source code. Requests should be sent by e-mail to sle_source_request@suse.com or as otherwise instructed at http://www.suse.com/download-linux/source-code.html. SUSE may charge a reasonable fee to recover distribution costs.

24 Legal Notices #

SUSE makes no representations or warranties with regard to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, SUSE reserves the right to revise this publication and to make changes to its content, at any time, without the obligation to notify any person or entity of such revisions or changes.

Further, SUSE makes no representations or warranties with regard to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, SUSE reserves the right to make changes to any and all parts of SUSE software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classifications to export, re-export, or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical/biological weaponry end uses. Refer to https://www.suse.com/company/legal/ for more information on exporting SUSE software. SUSE assumes no responsibility for your failure to obtain any necessary export approvals.

This release notes document is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC-BY-SA-4.0). You should have received a copy of the license along with this document. If not, see https://creativecommons.org/licenses/by-sa/4.0/.

SUSE has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at https://www.suse.com/company/legal/ and one or more additional patents or pending patent applications in the U.S. and other countries.

For SUSE trademarks, see SUSE Trademark and Service Mark list (https://www.suse.com/company/legal/). All third-party trademarks are the property of their respective owners.

SUSE CaaS Platform 4.2.7 Release Notes

1 About the Release Notes #

2 Changes in 4.2.10 #

2.1 Cilium Update #

2.2 Required Actions #

2.3 Bugs Fixed in 4.2.10 since 4.2.9 #

3 Changes in 4.2.9 #

3.1 CRI-O Update #

3.2 Required Actions #

3.3 Bugs Fixed in 4.2.9 since 4.2.8 #

4 Changes in 4.2.8 #

4.1 CRI-O Update #

4.2 Required Actions #

4.3 Bugs Fixed in 4.2.8 since 4.2.7 #

5 Changes in 4.2.7 #

5.1 CRI-O Update #

5.2 Required Actions #

5.3 Bugs Fixed in 4.2.7 since 4.2.6 #

6 Changes in 4.2.6 #

6.1 Kubernetes Update #

6.2 Required Actions #

6.3 Bugs Fixed in 4.2.6 since 4.2.5 #

7 Changes in 4.2.5 #

7.1 Skuba Update #

7.2 Required Actions #

7.3 Bugs Fixed in 4.2.5 since 4.2.4 #

8 Changes in 4.2.4 #

8.1 'helm3' Package Available #

8.2 Kubernetes Update #

8.3 kucero addon #

8.4 Required Actions #

8.5 Bugs Fixed in 4.2.4 since 4.2.3 #

8.6 Documentation Changes #

8.7 Known Issues #

9 Changes in 4.2.3 #

9.1 Required Actions #

9.2 Bugs Fixed in 4.2.3 since 4.2.2 #

9.3 Documentation Changes #

9.4 Known Issues #

9.4.1 sles15sp1 aws image is inactive #

10 Changes in 4.2.2 #

10.1 Required Actions #

10.2 Bugs Fixed in 4.2.2 since 4.2.0 #

10.3 Documentation Changes #

10.4 Known Issues #

10.4.1 sles15sp1 aws image is inactive #

11 Changes in 4.2.1 #

11.1 Cilium Update to 1.6.6 with Envoy #

11.2 skuba #

11.3 Metrics Server #

11.4 Cert Status Checker #

11.5 VSphere VCP #

11.6 Required Actions #

11.7 Bugs Fixed in 4.2.1 since 4.2.0 #

11.8 Documentation Changes #

11.9 Known Issues #

12 Changes in 4.2.0 #

12.1 Deprecations in 4.2.0 #

12.2 Kubernetes Update #

12.3 Addon Customization Persistence #

12.4 Skuba: Disabling firewalld #

12.5 Datastore for VMware #

12.6 Required Actions #

12.6.1 Kubernetes 1.17 #

12.6.2 Conmon and CRI-O #

12.6.3 Skuba #

12.6.4 Generate the kustomize Style Addon Configurations #

12.7 Bugs Fixed in 4.2.0 since 4.1.2 #

12.8 Documentation Changes #

12.9 Known Issues #

12.9.1 skuba-update Error: patterns-caasp-Node Conflicts with CRI-O Update #

13 Changes in 4.1.2 #

13.1 Deployment on AWS as Technology Preview #

13.2 Terraform Upgrade #

13.3 etcd Backup and Restore for Master Nodes Disaster Recovery #

13.4 Velero for Disaster Recovery #

13.5 Required Actions #

13.5.1 Upgrade Terraform Files and State #

13.5.2 Recommended Procedure #

Important

11.2 `skuba` #

12.4 Skuba: Disabling `firewalld` #

12.6.4 Generate the `kustomize` Style Addon Configurations #

12.9.1 `skuba-update` Error: `patterns-caasp-Node` Conflicts with CRI-O Update #

14.1.2 Fix Prometheus `kube-state-metrics` #