How SUSE builds its Enterprise Linux distribution – PART 5

This is the fifth blog of a series that provides insight into SUSE Linux Enterprise product development. You will get a first-hand overview of SUSE, the SLE products, what the engineering team does to tackle the challenges coming from the increasing pace of open source projects, and the new requirements from our customers, partners and business-related constraints.

How SUSE builds its Enterprise Linux distribution:

• PART 1 – Introduction
• PART 2 – Linux Distribution
• PART 3 – Release Management
• PART 4 – SUSE Linux Enterprise Schedules
• PART 5 – openSUSE and SLE

Whether you are a long-term SUSE customer, a new SUSE customer, a SUSE partner, or just an openSUSE fan or open source enthusiast, we hope you will find these blogs interesting, informative and helpful.

Linux Distributions Type

We have already covered Linux Distribution in a previous blog post, but something we didn’t discuss previously was the different types of Linux Distributions you can find in the wild. Now is the perfect time to explain the different types of Linux Distributions since you know a bit more about what is a Linux Distribution and our SLE Release Management and Schedules; this also perfectly fits with explaining the current relationship between openSUSE and SLE.

So first there is three main types of Linux Distributions defined by their release cycles and thus their targeted audience:

Rolling Release:

• Bleeding edge
• Release as soon as possible (CI/CD)
• Example: openSUSE Tumbleweed, ArchLinux, Manjaro, Gentoo

Regular:

• Release one to twice a year
• Updates the entire stack for each release
• Example: Ubuntu, Fedora, Debian

Long Term Support / Enterprise:

• Slow cadence (yearly more or less)
• Few things move between sub-releases, only major release brings “disruptive” changes
• Example: openSUSE Leap, Ubuntu LTS, SUSE Linux Enterprise Server, SUSE Linux Enterprise Desktop, RHEL, CentOS

This is the basis to understand the various relationship we have with SUSE Linux Enterprise and openSUSE because, as we will see in the next section, openSUSE Tumbleweed, openSUSE Leap and SUSE Linux Enterprise are bonded together.

openSUSE & SLE – Developed together

Here is a simple picture describing the relationship between openSUSE & SLE since the release date of Leap 15.0 (May 25, 2018):

The Factory Project is the development code stream all our distributions are based on, it is not a Linux distribution! It is the immediate source for openSUSE Tumbleweed and it eventually ends up in openSUSE Leap and the SUSE Linux Enterprise distributions. Put it simply, Factory is the development repository for openSUSE and SUSE in a CI/CD fashion!

Now you might wonder what’s the deal with the Factory and openSUSE Tumbleweed relationship. Well, it’s pretty simple! Factory is getting a constant flow of codes without any proper Quality Assurance apart from various code reviews (mostly done by bots), so the openSUSE community creates snapshots of Factory tested with openQA. When a snapshot is good, it becomes an update for openSUSE Tumbleweed; hence a rolling release.

Then further down the picture is the relation between openSUSE Tumbleweed, openSUSE Leap and SUSE Linux Enterprise.

Based on our joint schedule, openSUSE Leap and SLE have a predictable release time frame: a release every 12 months and a 6 months support overlap for the former and new release, thus when the time is ready a snapshot of openSUSE Tumbleweed is made and both openSUSE and SLE will use this snapshot to create our next distributions versions.

With this picture, we are not talking about our distribution per se yet, it’s only a pool of packages sources that we will use to build our respective distribution. But before going into how it’s built, note that it’s a simplified view because of course, there is always some back and forth between for instance openSUSE Leap/SLE and openSUSE Tumbleweed; it’s not just a one-way sync because during the development phase of our distributions, bugs are found and of course fixes are submitted back to Factory so openSUSE Tumbleweed also receives fixes from the process. For the sake of simplifying the picture we did not add these contributions as arrows.

Also at SUSE, Open source is in our genes so we have always contributed to openSUSE but, since 2017, our SUSE Release Team had enforce a rule called “Factory First Policy“, which force code submissions for SLE to be pushed to Factory first before it lands in SLE. This is a continuation of the “Upstream First” principle on the distribution level. It reduces maintenance effort and leverages the community.

When a SUSE internal code submission is sent to SLE15, an automated check will ensure similar submission was done to Factory, if not the submission will be automatically rejected and will require the SUSE Release Team to take a closer look and request the code change to be submitted to Factory. With this Factory First Policy, we are making sure that any SLE development is pushed to openSUSE even before it’s accepted in SLE!

How we built openSUSE & SLE so far

So let’s talk about how we technically built openSUSE & SLE distributions until openSUSE Leap 15.2 and SUSE Linux Enterprise 15 Service Pack 2.

The top of the picture should be familiar to you, and as we said in the last section, we use the same pool of packages sources to build openSUSE Leap and SUSE Linux Enterprise Server.

This is because openSUSE Leap and SLE wanted to

1. directly share a core set of packages list (blue diamond on top),
2. have “extra” packages (green “V” shapes) that can be updated more frequently or having different support level,

For openSUSE it’s pretty simple, the bigger diamond (blue one + green V) represents the entire openSUSE Leap distribution with all it’s official packages.

For SUSE Linux Enterprise, our official distributions and packages are only the blue diamond on top! But the rest of the green “V” will be available in Package Hub. Package Hub is our community repository made by our community for SUSE Linux Enterprise Server and Desktop, but of course SUSE does not directly support those packages, it’s community supported.

The important part here is that openSUSE Leap 15.2 and SLE 15 SP2 use the same packages sources and share the same packages lists BUT we didn’t use the same binaries rpm!

openSUSE and SLE in the near future

So we just saw the “weirdest chameleon symbiosis” found in nature, but how can we make it better? It’s easy, by simplifying the big picture:

The previous scheme (“How we built openSUSE & SLE so far” ) was used by at least our last 3 releases, but SUSE felt like we could move forward and do more, so we have kickstarted the Closing the Leap Gap proposal to the openSUSE community during the SLE 15 SP2 Development phase. To make a long story short, the proposal was to include pre-built binaries from SLE in addition to the sources we were already providing to increase compatibility and to leverage synergies. For more details on all the aspect of this proposal, we can not emphasis enough reading of the openSUSE FAQ page.

The change to our relations are made because we want a smoother migration path between SLE and openSUSE Leap but also to look for more direct collaboration with the openSUSE community. Therefore SUSE is also making easier for the community to contribute directly to openSUSE and SLE via new dedicated channels. So the community is still able to shape and submit changes to the next distribution version with this new setup.

Please check-out the following links for more information:

• https://news.opensuse.org/2020/04/29/Discuss-Define-and-be-Transparent-with-the-openSUSE-Community/
• https://en.opensuse.org/Portal:Jump/Policy/CommunitySLEChangeRequests
• https://en.opensuse.org/Portal:Jump:OBS:SRMirroring
• https://en.opensuse.org/Portal:15.3/Features:Identicality

The ultimate goal on a project perspective is to make an healthy and self-sufficient ecosystem, and on a distribution level to have a good balance between an environment suitable for production and for innovation.

As you can see, the relationship between openSUSE and SLE is not complicated per se, but it’s true that we have chosen our very own symbiosis that allows to create boundaries between a community rolling release distribution (openSUSE Tumbleweed), a community LTS distribution (openSUSE Leap) and an enterprise distribution (SLE). And with Closing the Leap Gap project, what we want to achieve is to keep improving the efficiency of contribution to and from the community and the enterprise side.

The future looks good.

We share more than code

Last but not least, the openSUSE community and SUSE share way more than just code! But if we stick with the software development area, we have to talk about how openSUSE and SLE are build and test during their bonded development phase.

So next we will talk about some of the underlying processes glueing everything together but also about the great tool we are using: Open Build Service (build) and openQA (test).

Further Readings/Videos:

• https://en.opensuse.org/Portal:15.3
• Youtube: openSUSE Leap 15.3 Kickoff on Nov. 4, 2020
• Youtube: openSUSE Leap Next
• Youtube: Why Should Companies be interested in the next version openSUSE Leap