Back to Release Notes for SUSE products › Release Notes
Show Contents: Release Notes
Top
  1. 1 SUSE CaaS Platform 3
  2. 2 Advantages of SUSE CaaS Platform
  3. 3 Support and Life Cycle
  4. 4 Requirements
  5. 5 New Features
  6. 6 Technology Previews
  7. 7 Product Updates
  8. 8 Documentation and Other Information
  9. 9 How to Obtain Source Code
  10. 10 Support Statement for SUSE CaaS Platform
  11. 11 Technical Information
  12. 12 Legal Notices
SUSE CaaS Platform 3

Release Notes

This document provides guidance and an overview of high-level general features and updates for SUSE CaaS Platform 3. It describes the capabilities and limitations of SUSE CaaS Platform 3.

These release notes are updated periodically. The latest version is always available at https://www.suse.com/releasenotes. General documentation can be found at: https://www.suse.com/documentation/suse-caasp/.

Publication Date: 2019-11-12, Version: 3.0.20191111
1 SUSE CaaS Platform 3
1.1 SUSE MicroOS
1.2 Features
1.3 Components
1.4 File System
2 Advantages of SUSE CaaS Platform
3 Support and Life Cycle
4 Requirements
4.1 Cluster Requirements
4.2 Supported Environments
4.3 Minimum Node Specification
4.4 Container Data Storage
4.5 Cluster Networking Requirements
5 New Features
6 Technology Previews
6.1 CRI-O Has Been Added
7 Product Updates
7.1 Flannel Container Security Fix Needs Manual Setup Adjustments
8 Documentation and Other Information
8.1 Available on the Product Media
8.2 Externally Provided Documentation
9 How to Obtain Source Code
10 Support Statement for SUSE CaaS Platform
11 Technical Information
11.1 Kernel Limits
11.2 File Systems
12 Legal Notices

1 SUSE CaaS Platform 3

SUSE CaaS Platform (Container as a Service Platform) is an integrated software platform which automates the process of building, managing and upgrading of Kubernetes clusters. It combines the benefits of an enterprise-ready operating system with the agility of an orchestration platform for containerized applications.

Kubernetes (https://kubernetes.io/) is the industry-standard orchestration framework for optimizing the use of containers. It automates the deployment, scaling, and operations of pods of containers across multiple nodes in a cluster.

SUSE CaaS Platform 3 is shipped with Kubernetes version 1.10.11.

1.1 SUSE MicroOS

The basis of SUSE CaaS Platform is SUSE MicroOS (https://en.opensuse.org/Kubic:MicroOS).

SUSE MicroOS is a minimalist operating system based on SUSE Linux Enterprise 12 SP3, dedicated to hosting containers. SUSE MicroOS inherits the benefits of SUSE Linux Enterprise in the form of a smaller, simpler, more efficient and robust operating system, optimized for large, clustered deployments.

SUSE MicroOS also features an atomic, transactional update mechanism, making the system more resilient against software-update-related problems.

1.2 Features

SUSE CaaS Platform is delivered as a complete, integrated software stack. The result is a complete, fully-configured Kubernetes cluster, ready to host containers.

  • Supports installation directly onto hardware and into virtual machines, as well as public cloud deployments.

  • Automated installation with AutoYaST: after the first cluster node has been installed, it can control the installation of the other nodes.

  • Pre-installed VM images are also available for deployment into existing open-source hypervisor infrastructure.

  • Sophisticated copy-on-write-based Btrfs file system, with snapshots and fully transactional updates with rollback.

  • A web-based dashboard for cluster management.

  • The included version of Kubernetes supports role-based access control (RBAC).

1.3 Components

A SUSE CaaS Platform cluster consists of four or more nodes running SUSE MicroOS. Each node can host multiple pods of containers. The SUSE CaaS Platform stack consists, among others, of the following open-source tools:

  • Docker Open Source.  This is the leading open-source format for application containers. It is fully supported by SUSE.

    For more information, see https://www.docker.com/.

  • Flannel (CNI).  Flannel is a network fabric for containers, supplied as a plugin for CNI. The Container Network Interface provides the virtual network for inter-container communications.

    For more information, see https://github.com/coreos/flannel and https://github.com/containernetworking/cni.

  • Velum.  Web-based dashboard for graphical, point-and-click management of the entire cluster.

    For more information, see https://github.com/kubic-project/velum.

  • cloud-init.  Individual nodes of the cluster are provisioned by an enhanced version of cloud-init which can now configure Zypp repositories and read from local directories. This tool is very flexible and already well-known by administrators in cloud environments.

    For more information, see https://cloud-init.io/.

  • Salt.  The software running on the individual nodes of the cluster is managed by Salt.

    For more information, see https://github.com/saltstack/salt.

  • etcd.  etcd is a key/value store used by the Kubernetes API server to keep the state of the cluster.

    For more information, see https://coreos.com/etcd/.

  • Dex.  User authentication is handled by Dex which includes role-based access control, allowing customers to restrict what actions are available to different cluster users.

    More information: https://github.com/coreos/dex.

1.4 File System

SUSE MicroOS comes with a novel file system configuration based on Btrfs and OverlayFS, designed for robustness and to allow transactional updates.

  • A read-only root file system protects the base operating system from damage or corruption.

  • OverlayFS is used for /etc (for cloud-init and Salt).

  • Subvolumes for data storage are read-write.

  • Copy-on-write snapshots permit in-place software updates with roll-back if required.

2 Advantages of SUSE CaaS Platform

SUSE CaaS Platform allows you to provision, manage, and scale container-based applications. SUSE offers an application development and hosting platform for containers that automates the tedious management tasks. This allows you to focus on development and writing apps that meet business goals.

The main benefits of SUSE CaaS Platform are:

  • Enable DevOps and Microservices Applications:  Develop, deploy, and automate modular, reliable, and serviceable applications across infrastructure, regardless of the application's architecture.

  • Enterprise-Grade Security and Scalability:  The foundation of SUSE CaaS Platform is our SUSE Linux Enterprise platform which provides a stable and reliable environment for enterprise applications.

  • Run Everywhere:  You can quickly and intelligently respond to demand across private and public clouds. SUSE CaaS Platform helps you manage peak demand without downtime or manual intervention.

  • Accelerate Business Innovation:  Go faster from concept to production. Give developers and operations teams the tools they need to iterate faster and reduce time between application releases.

3 Support and Life Cycle

SUSE CaaS Platform is backed by award-winning support from SUSE, an established technology leader with a proven history of delivering enterprise-quality support services.

The support for version SUSE CaaS Platform 3 ends with the release of version 4. We offer rolling updates allowing customers to upgrade the system via a maintenance update. When releasing version 4, SUSE will offer maintenance updates including packages enabling our customers to move to version 4 via a simple update.

For more information, check our Support Policy page https://www.suse.com/support/programs/subscriptions/?id=SUSE_CaaS_Platform.

4 Requirements

This section lists requirements that SUSE CaaS Platform has for its host environment and the underlying hardware.

4.1 Cluster Requirements

SUSE CaaS Platform 3 is a cluster operating system and requires a minimum of 4 nodes: 1 admin node, 1 master node and 2 worker nodes.

Important
Important: 3-Node Clusters

For proof-of-concept and test deployments, you can bootstrap a 3-node cluster with a single worker node, but this configuration is not supported and should not be used in production.

To improve performance and reliability, additional master nodes may be added, so long as there is an odd number of master nodes.

There is no limit on the number of worker nodes except the maximum cluster size.

SUSE currently supports clusters of up to 150 nodes.

4.2 Supported Environments

Regarding deployment scenarios, SUSE supports SUSE CaaS Platform running in the following environments:

  • SUSE CaaS Platform only supports x86_64 hardware.

  • Aside from this, the same platforms as SUSE Linux Enterprise 12 SP3 are supported.

  • On bare metal: any server hardware certified for SUSE Linux Enterprise Server.

  • Virtualized—running under the following hypervisors:

    • KVM

      • on SUSE Linux Enterprise 11 SP4

      • on SUSE Linux Enterprise 12 SP1

      • on SUSE Linux Enterprise 12 SP2

      • on SUSE Linux Enterprise 12 SP3

    • Xen

      • same host platforms as for KVM

      • full virtualization

      • paravirtualization

      • Citrix XenServer 6.5

    • VMware

      • ESX 5.5

      • ESXi 6.0

      • ESXi 6.5

    • Hyper-V

      • Windows Server 2008 SP2+

      • Windows Server 2008 R2 SP1+

      • Windows Server 2012+

      • Windows Server 2012 R2+

      • Windows Server 2016

    • Oracle VM 3.3

  • Private and Public Cloud Environments

    • SUSE OpenStack Cloud 7

    • Amazon Web Services

    • Microsoft Azure

    • Google Cloud Platform (Google Compute Engine)

4.3 Minimum Node Specification

Each node in the cluster must meet the following minimum specifications.

  • Minimum RAM: 8 GB.

    Note
    Note: Swap Partitions

    Kubernetes does not support swap and any existing swap partition will be disabled during installation. Although it may be possible to install SUSE CaaS Platform on nodes with less RAM, if the operating system runs out of available memory, the cluster may fail.

  • Minimum disk size: 40 GB (as Btrfs with snapshots) for the root file system. Additional disk space is recommended for containers and container images.

  • Kernel limits are documented in Section 11.1, “Kernel Limits”.

  • Installation using VNC is not supported.

4.4 Container Data Storage

Storage can be provided using:

  • SUSE Enterprise Storage.

  • NFS.

  • hostpath.

    Note
    Note: hostpath Is Deprecated

    hostpath storage is still supported, but its use is now discouraged. By default, it is disabled by PodSecurityPolicies.

4.5 Cluster Networking Requirements

All the nodes on the cluster must be on a the same network and be able to communicate directly with one another.

The admin node and the Kubernetes API master must have valid Fully-Qualified Domain Names (FQDNs), which can be resolved both by other nodes and from other networks which need to access the cluster.

On the same network, a separate computer with a Web browser is required in order to complete bootstrap of the cluster.

5 New Features

SUSE CaaS Platform 3 includes the following new features:

  • Optimize cluster configuration with expanded data center integration and cluster reconfiguration options.

  • A new SUSE toolchain module also allows tuning the SUSE MicroOS container operating system to support custom hardware configuration needs. For example, you can now install additional packages required to run a monitoring agent or other custom software.

  • Transform your start-up cluster into a highly available environment. With new cluster reconfiguration capabilities, you can switch from a single-master to a multi-master environment, or vice versa, to accommodate changing needs.

  • Manage container images more efficiently and securely with a local container registry.

  • Download a container image from an external registry once, then save a copy in your local registry for all your cluster nodes to share. This can improve security and increase performance every time a cluster node pulls an image from the local registry: Nodes only connect to an internal proxy instead of an external registry and download from a local cache instead of a remote server.

  • For still greater security, disconnect from external registries and use only trusted images that you have loaded into your local registry.

  • Simplify deployment and management of long running workloads through the Apps Workloads API. Promoted to “stable” in upstream Kubernetes 1.9 code, the Apps Workloads API is now supported by SUSE. This API facilitates orchestration (self-healing, scaling, updates, termination) of common types of workloads.

  • As a technology preview, SUSE CaaS Platform 3 includes the lightweight CRI-O container runtime, designed specifically for Kubernetes. For more information, see Section 6.1, “CRI-O Has Been Added”.

6 Technology Previews

Technology previews are packages, stacks, or features delivered by SUSE which are not supported. They may be functionally incomplete, unstable or in other ways not suitable for production use. They are included for your convenience and give you a chance to test new technologies within an enterprise environment.

Whether a technology preview becomes a fully supported technology later depends on customer and market feedback. Technology previews can be dropped at any time and SUSE does not commit to providing a supported version of such technologies in the future.

Give your SUSE representative feedback, including your experience and use case.

6.1 CRI-O Has Been Added

As a technology preview, SUSE CaaS Platform 3 includes the CRI-O container runtime in addition to Docker Open Source. CRI-O is an implementation of CRI, the Container Runtime Interface and was designed specifically for Kubernetes. The development focus of CRI-O is creating a lightweight and architecturally simple container engine that is also stable and secure. Like Docker Open Source, it supports Open Container Initiative (OCI) images.

During the installation, you can choose whether to set up the complete cluster with the Docker Open Source container engine, the default, or with the CRI-O container engine. As both the Docker Open Source engine and CRI-O use the same runtime component, called runC, usage of a container engine is transparent, no changes to workloads and images are needed.

For more information about CRI-O, see http://cri-o.io/.

7 Product Updates

7.1 Flannel Container Security Fix Needs Manual Setup Adjustments

In November 2019, a security update fixing Flannel containers running in privileged mode was shipped with the package kubernetes-salt version 3.0.0+git_r999_f540bd3-3.77.1. To apply the fix, you need to make changes in the kube-flannel DaemonSet of the pods. For that, the pods must be re-created. During the re-creation of the pods, the workloads will not have network connectivity for up to 1 minute.

In the majority of the cases, the workloads running on top of Kubernetes can handle a short network interruption. If that is the case for your workloads, you can skip the cordon and drain steps in the following procedure.

If even a short network interruption is undesired, make sure to cordon and drain the nodes from their pods as described below.

The following procedure demonstates how to update the kube-flannel pod on node vm152201, but it must be applied to all nodes in the cluster. You can do so in parallel or serialized.

  1. (Optional) List the running pods on node vm152201. 

    kubectl get po --all-namespaces --field-selector spec.nodeName=vm152201
NAMESPACE     NAME                              READY   STATUS   RESTARTS   AGE
default       nginx-deployment-76bfd5cc-8x2fs   1/1     Running  1          1h
default       nginx-deployment-76bfd5cc-b7ztt   1/1     Running  1          17m
default       nginx-deployment-76bfd5cc-pg72w   1/1     Running  1          1h
default       nginx-deployment-76bfd5cc-vpscp   1/1     Running  1          1h
kube-system   haproxy-vm152201                  1/1     Running  2          2d
kube-system   kube-dns-7459d67f57-bjpq4         3/3     Running  3          2d
kube-system   kube-dns-7459d67f57-n55xk         3/3     Running  3          2d
kube-system   kube-flannel-q2p4s                1/1     Running  3          2d
kube-system   tiller-deploy-79b5d8d474-snwmz    1/1     Running  1          2d

  2. (Optional) Verify that the pod still has the old configuration and is running privileged: 

    kubectl -n kube-system get po \
  kube-flannel-q2p4s -ojsonpath='{.spec.containers[0].securityContext.privileged}'
true

  3. Mark node vm152201 as unschedulable (cordon): 

    kubectl cordon vm152201
node "vm152201" cordoned

  4. Drain the node vm152201 in preparation for maintenance: 

    kubectl drain vm152201 --grace-period=300 --force --ignore-daemonsets
node "vm152201" already cordoned
WARNING: Ignoring DaemonSet-managed pods: kube-flannel-q2p4s
pod "tiller-deploy-79b5d8d474-snwmz" evicted
pod "nginx-deployment-76bfd5cc-vpscp" evicted
pod "nginx-deployment-76bfd5cc-8x2fs" evicted
pod "nginx-deployment-76bfd5cc-pg72w" evicted
pod "nginx-deployment-76bfd5cc-b7ztt" evicted
pod "kube-dns-7459d67f57-bjpq4" evicted
pod "kube-dns-7459d67f57-n55xk" evicted
node "vm152201" drained

    The above command allows 5 minutes (300 seconds) for each pod to be evicted. If your workloads require more, set the --grace-period to a higher value.

  5. Delete the kube-flannel pod: 

    kubectl -n kube-system delete po kube-flannel-q2p4s
pod "kube-flannel-q2p4s" deleted

  6. Wait until the pod is terminated and the new one is ready: 

    watch kubectl -n kube-system get po \
--selector=k8s-app=flannel --field-selector spec.nodeName=vm152201
NAME                 READY     STATUS    RESTARTS   AGE
kube-flannel-k88rh   1/1       Running   0          10s

  7. Verify the changes in the new pod kube-flannel-k88rh. To do so, verify that the pod is no longer running in privileged mode: 

    kubectl -n kube-system get po \
kube-flannel-k88rh -ojsonpath='{.spec.containers[0].securityContext.privileged}'
false

  8. If you cordoned the node earlier, scheduling will remain disabled on the node: 

    kubectl get nodes
NAME       STATUS                     ROLES     AGE       VERSION
vm152192   Ready                      master    2d        v1.10.11
vm152194   Ready                      master    2d        v1.10.11
vm153221   Ready                      master    2d        v1.10.11
vm152201   Ready,SchedulingDisabled   none      2d        v1.10.11
vm154205   Ready                      none      2d        v1.10.11
vm154207   Ready                      none      2d        v1.10.11

  9. To mark the node vm152201 as schedulable (uncordon), use: 

    kubectl uncordon vm152201
node "vm152201" uncordoned

  10. Scheduling is now enabled on the node again: 

    kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
vm152192   Ready     master    2d        v1.10.11
vm152194   Ready     master    2d        v1.10.11
vm153221   Ready     master    2d        v1.10.11
vm152201   Ready     none      2d        v1.10.11
vm154205   Ready     none      2d        v1.10.11
vm154207   Ready     none      2d        v1.10.11

8 Documentation and Other Information

8.1 Available on the Product Media

  • Read the READMEs on the media.

  • Get the detailed change log information about a particular package from the RPM (where FILENAME is the name of the RPM): 

    rpm --changelog -qp FILENAME.rpm

  • Check the ChangeLog file in the top level of the media for a chronological log of all changes made to the updated packages.

  • Find more information in the docu directory of the media of SUSE CaaS Platform 3.

  • The most recent version is always available online at https://www.suse.com/releasenotes. Some entries may be listed twice, if they are important and belong to more than one section.

8.2 Externally Provided Documentation

9 How to Obtain Source Code

This SUSE product includes materials licensed to SUSE under the GNU General Public License (GPL). The GPL requires SUSE to provide the source code that corresponds to the GPL-licensed material. The source code is available for download at https://www.suse.com/download-linux/source-code.html.

Also, for up to three years after distribution of the SUSE product, upon request, SUSE will mail a copy of the source code. Requests should be sent by e-mail to mailto:sle_source_request@suse.com or as otherwise instructed at https://www.suse.com/download-linux/source-code.html. SUSE may charge a reasonable fee to recover distribution costs.

10 Support Statement for SUSE CaaS Platform

To receive support, you need an appropriate subscription with SUSE. For more information, see https://www.suse.com/support/programs/subscriptions/?id=SUSE_CaaS_Platform.

The following definitions apply:

L1

Problem determination, which means technical support designed to provide compatibility information, usage support, ongoing maintenance, information gathering and basic troubleshooting using available documentation.

L2

Problem isolation, which means technical support designed to analyze data, reproduce customer problems, isolate problem area and provide a resolution for problems not resolved by Level 1 or prepare for Level 3.

L3

Problem resolution, which means technical support designed to resolve problems by engaging engineering to resolve product defects which have been identified by Level 2 Support.

For contracted customers and partners, SUSE CaaS Platform v3 is delivered with L3 support for all packages, except the following:

  • Technology Previews

  • sound, graphics, fonts and artwork

  • packages that require an additional customer contract

  • packages provided as part of the Software Development Kit (SDK)

SUSE will only support the usage of original (that is, unchanged and un-recompiled) packages.

11 Technical Information

This section contains information about system limits, several technical changes and enhancements for the experienced user.

When talking about CPUs, we use the following terminology:

CPU Socket

The visible physical entity, as it is typically mounted to a mainboard or an equivalent.

CPU Core

The (usually not visible) physical entity as reported by the CPU vendor.

Logical CPU

This is what the Linux Kernel recognizes as a “CPU”.

We avoid the word “thread” (which is sometimes used), as the word “thread” would also become ambiguous subsequently.

Virtual CPU

A logical CPU as seen from within a Virtual Machine.

11.1 Kernel Limits

This table summarizes the various limits which exist in our recent kernels and utilities (if related) for SUSE CaaS Platform 3.

SUSE CaaS Platform 3 (Linux 4.4)Intel 64/AMD64 (x86-64)

CPU bits

64

Maximum number of logical CPUs

8192

Maximum amount of RAM (theoretical/certified)

> 1 PiB/64 TiB

Maximum amount of user space/kernel space

128 TiB/128 TiB

Maximum amount of swap space

Up to 29 * 64 GB

Maximum number of processes

1048576

Maximum number of threads per process

Upper limit depends on memory and other parameters (tested with more than 120,000).

Maximum size per block device

Up to 8 EiB

FD_SETSIZE

1024

11.2 File Systems

SUSE CaaS Platform 3 exclusively supports the file system types Btrfs and OverlayFS. The root file system is a read-only Btrfs which enables transactional updates.

The following table lists supported and unsupported Btrfs features across multiple SUSE CaaS Platform versions.

+ supported
– unsupported

FeatureBtrfs on SUSE CaaS Platform 3

Offline extend/shrink

+ / +

Online extend/shrink

+ / +

Inode allocation map

B-tree

Sparse files

+

Tail packing

+

ExtAttr/ACLs

+ / +

Quotas

+

Dump/restore

–

Block size default

4 KiB

Maximum file system size

16 EiB

Maximum file size

16 EiB

Data/metadata journaling

N/A *

Copy on Write+
Snapshots/Subvolumes+
Metadata Integrity+
Data Integrity+
Online Metadata Scrubbing+
Automatic Defragmentation–
Manual Defragmentation+
In-band Deduplication–
Out-of-band Deduplication+
Quota Groups+
Metadata Duplication+
Multiple Devices+
RAID 0+
RAID 1+
RAID 10+
RAID 5–
RAID 6–
Hot Add/Remove+
Device Replace–
Seeding Devices–
Compression+
Big Metadata Blocks+
Skinny Metadata+
Send Without File Data+
Send/Receive+
Inode Cache–
Fallocate with Hole Punch+

* Btrfs is a copy-on-write file system. Instead of journaling changes before writing them in-place, it writes them to a new location and then links the new location in. Until the last write, the changes are not “committed”. Because of the nature of the file system, quotas are implemented based on subvolumes (qgroups).

The maximum file size above can be larger than the file system's actual size because of usage of sparse blocks. The numbers in the above table assume that the file systems are using 4 KiB block size. When using different block sizes, the results are different, but 4 KiB reflects the most common standard.

In this document: 1024 Bytes = 1 KiB; 1024 KiB = 1 MiB; 1024 MiB = 1 GiB; 1024 GiB = 1 TiB; 1024 TiB = 1 PiB; 1024 PiB = 1 EiB. See also http://physics.nist.gov/cuu/Units/binary.html.

For more information, also see: https://www.suse.com/products/caas-platform/technical-information/#FileSystem

Print this page

© 2019 SUSE