sapconf – A way to prepare a SLES system for SAP workload – Part 2

This is the second article about sapconf and it will cover the configuration.

NOTE: For further documentation check SAP note 1275776 and the man pages shipped with sapconf.

Configuration

Before I show you some details, I have to talk about the make-up of sapconf.

In the past, sapconf was a simple shell script. With SLES 12 SP1 tuned came along and both became a merry couple, so nowadays one part of the configuration is done by tuned itself and another part is done by a shell script executed by tuned.

Tuned is a central component of sapconf.

sapconf 4 → sapconf 5

With sapconf 5 this has changed! We have removed tuned completely!

Sounds complicated? Maybe a little picture helps:

sapconf 4 and tuned

sapconf 4 → sapconf 5

Here a picture for sapconf 5:

sapconf 5 and tuned

What you have to take with you, is that the configuration is split into two parts:

• A configuration file for tuned and
• a configuration file for the (sapconf) script (which is started by tuned).

sapconf 4 → sapconf 5

The entire configuration is now done in the sapconf 5 configuration files.

Who has worked with tuned already, knows that the basic element of tuned is a so called profile. To list all available profiles, run tuned-adm list:

sles12-sp3:~ # tuned-adm list
Available profiles:
- balanced
- desktop
- latency-performance
- network-latency
- network-throughput
- powersave
- sap-ase
- sap-bobj
- sap-hana
- sap-netweaver
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: sap-netweaver

The profiles ‘sap-ase’, ‘sap-bobj’, ‘sap-hana’ and ‘sap-netweaver’ are shipped with sapconf, the others with tuned. There might be even more profiles coming from different packages (like saptune).
The sapconf installation also has set ‘sap-netweaver’, because no other profile was configured.

NOTE: As mentioned in the previous article, sapconf will set the tuned profile on installation only if it is empty.

The re-work of sapconf included a unification process of these profiles. Sapconf shall become a simple all-purpose tool again. Who wants more intervention should have a look at saptune.

As a result in SLES 15 only one profile will be shipped to rule them all:’sapconf’.
Why not doing the same in SLES 12, you ask? Well, because bigger changes are only possible with a new SLES release to prevent a regression. The profiles have to stay, but we can make them equal. Hence in SLES 12, the profiles ‘sap-hana’ and ‘sap-netweaver’ are identical and you can choose either of it.
The profiles ‘sap-ase’ and ‘sap-bobj’ still work, but have been left out in the rework process for now, since they have been used rarely. Some parameters are even still hard-coded in the script and cannot be changed.
With SP4 they might become identical to ‘sap-netweaver’ too, but this has not been decided yet.
Nevertheless you should move away from ‘sap-ase’ and ‘sap-bobj’.

sapconf 4 → sapconf 5

You will not find any tuned profiles for sapconf 5.
Profiles still exist in SLES 12, but these are sapconf profiles not tuned profiles and handled by sapconf 5 internally. In SLES 15 profiles does not exist anymore!

In case you want to change the profile, just run tuned-adm profile .

WARNING: The profiles ‘sap-ase’ and ‘sap-bobj’ should not be used anymore!
Instead choose ‘sap-hana’ or ‘sap-netweaver’ which are identical.
In SLE 15 only one profile ‘sapconf’ will be shipped.

After this little detour to tuned and the background of sapconf, back to our configuration.

sapconf 4 → sapconf 5

If you want to change the profile in sapconf 5 (only SLES 12), please run:

sapconf stop && sapconf <new_profile>

The profiles are named: netweaver, hana, bobj and ase.
The statement above about the profiles bobj and ase still applies. They’re not maintained anymore and support for them is gone in SLES 15.

During installation we have got some information about configuration files, which I have skipped. Now it is time to address it.

... 
Updating /etc/sysconfig/sapnote-1680803... 
Updating /etc/sysconfig/sapconf... 
...

The file /etc/sysconfig/sapnote-1680803 belongs to the ASE profile (check man /etc/sysconfig/sapnote-1680803), which I will ignore.

One of both main configuration files is /etc/sysconfig/sapconf.
Since sapconf uses tuned, I have to look at the tuned configuration as well. But first things first.

Let me grep thru /etc/sysconfig/sapconf to see what parameters we have and what the default values are:

sles12-sp3:~ # grep '^#*[A-Z]' /etc/sysconfig/sapconf
VSZ_TMPFS_PERCENT=75
SHMALL=1152921504606846720
SHMMAX=18446744073709551615
SEMMSL=32000
SEMMNS=1024000000
SEMOPM=500
SEMMNI=32000
LIMIT_1="@sapsys soft nofile 65536"
LIMIT_2="@sapsys hard nofile 65536"
LIMIT_3="@sdba soft nofile 65536"
LIMIT_4="@sdba hard nofile 65536"
LIMIT_5="@dba soft nofile 65536"
LIMIT_6="@dba hard nofile 65536"
MAX_MAP_COUNT=2147483647
SHMMNI=32768
DIRTY_BYTES=629145600
DIRTY_BG_BYTES=314572800
TCP_SLOW_START=0
KSM=0
NUMA_BALANCING=0
THP=never
ENABLE_PAGECACHE_LIMIT="no"
#PAGECACHE_LIMIT_MB=""
#PAGECACHE_LIMIT_IGNORE_DIRTY=""
#SAPCONF_END=""

sapconf 4 → sapconf 5

With the removal of tuned in sapconf 5 you will see some additional variables:

PERF_BIAS=
GOVERNOR=
FORCE_LATENCY=70
MIN_PERF_PCT=
IO_SCHEDULER="noop none"

NOTE: To learn more about each variable, browse thru /etc/sysconfig/sapconf. Each variable is preluded by a comment section that explains the background and the origin (SAP note).

Some variables are commented out. The reason for this is, that there are no sane default values. Those variables have to be set only, if it makes sense.

WARNING: Do not comment out a variable unless checking the effect with the sapconf log. Future versions will support this and leave the system setting alone. The current version does not do this in every case.

To demonstrate how to change a value, I will change TCP_SLOW_START from 0 to 1.

First I will check the current value to verify that it is really 0 as configured. The comment section tells me where to look:

sles12-sp3:~ # less /etc/sysconfig/sapconf 
... 
# net.ipv4.tcp_slow_start_after_idle 
# If enabled (=1), provide RFC 2861 behavior and time out the congestion 
# window after an idle period. An idle period is defined as the current ... TCP_SLOW_START=0 
... 
sles12-sp3:~ # sysctl net.ipv4.tcp_slow_start_after_idle 
net.ipv4.tcp_slow_start_after_idle = 0

Now I use my favorite editor and change TCP_SLOW_START=0 to TCP_SLOW_START=1 and restart sapconf:

sles12-sp3:~ # vi /etc/sysconfig/sapconf 
... 
sles12-sp3:~ # systemctl restart sapconf.service 
sles12-sp3:~ # sysctl net.ipv4.tcp_slow_start_after_idle 
net.ipv4.tcp_slow_start_after_idle = 1

NOTE: Earlier versions of sapconf did nothing on a restart. The new version behaves as expected and reloads the configuration!

The parameter has been set to 1 as configured.

Setting net.ipv4.tcp_slow_start_after_idle is also noted in sapconfs log file /var/log/sapconf.log:

sles12-sp3:~ # tail /var/log/sapconf.log 
... 
2018-05-14 17:17:28+02:00 Leaving net.ipv4.tcp_slow_start_after_idle unchanged at 1 
...

Wait a minute… Leaving it unchanged at 1? What’s that suppose to mean?

sles12-sp3:~ # grep net.ipv4.tcp_slow_start_after_idle /var/log/sapconf.log
2018-05-14 16:11:47+02:00   Change net.ipv4.tcp_slow_start_after_idle from 1 to 0
2018-05-14 17:17:28+02:00   Restoring net.ipv4.tcp_slow_start_after_idle=1
2018-05-14 17:17:28+02:00   Leaving net.ipv4.tcp_slow_start_after_idle unchanged at 1

To understand the log messages you have to know how sapconf works.
On start it saves the current values of all parameters. When stopped all values will be reverted back.
This is behavior adopted from tuned. On profile entry tuned will save the current state and revert it when a profile is left.
If you have read the previous article carefully, you might remember that a sapconf restart does nothing else than stopping and starting tuned. This means leaving and entering a (sapconf) profile, which in turn calls the sapconf script.sh with stop and start as parameters (see the picture above).

With this in mind the lines make sense.

sapconf 4 → sapconf 5

Even with tuned removed, sapconf 5 still behaves the same. It saves the current values on start and restores them on stop.

I did the change of the configuration at 17:16. At this time the value of net.ipv4.tcp_slow_start_after_idle was 0.
Than I restarted sapconf (17:17), which means a stop followed by a start of tuned and therefore the sapconf script.
At first sapconf restored the value back to 1, because the value was 1 when sapconf has been started (it is the default on SLES 12).
Than sapconf applied the new configuration. Because the value was already 1, it had nothing to do.

NOTE: To verify what sapconf does, check the log file /var/log/sapconf.log.
(But do not forget about the tuned log /var/log/tuned/tuned.log.)

sapconf 4 → sapconf 5

With sapconf 5 only /var/log/sapconf.log is used.
Of cause you will find entries in the system log when starting and stopping the sapconf service.

As said before, there is a second configuration due to the use of tuned.

sapconf 4 → sapconf 5

If you are using sapconf 5 – and you should – you can skip the rest of the article. It is about the tuned part which does not apply anymore.

All profiles have their configuration in /usr/lib/tuned/. At least a tuned.conf can be found there.
The directory /usr/lib/tuned/sap-netweaver contains a configuration file tuned.conf (Yes, that’s it!) and a script script.sh.

sles12-sp3:~ # ls /usr/lib/tuned/sap-netweaver/
script.sh  tuned.conf

The only file you should care about is tuned.conf. This is the second part of the sapconf configuration.
The script script.sh contains the logic, to save and restore the values and apply the configuration in /etc/sysconfig/sapconf. It should not be touched.

Here the tuned.conf:

sles12-sp3:~ # cat /usr/lib/tuned/sap-netweaver/tuned.conf
#
# basic tuned configuration for SAP NetWeaver, SAP HANA and HANA based products.
# Additional parameters are tuned by script.sh
# For a brief explanation of the parameters, please have a look at the end of
# the central sapconf configuration file /etc/sysconfig/sapconf
#
[main]
summary=Optimize for SAP NetWeaver, SAP HANA and HANA based products

[cpu]
 #governor = performance
 #energy_perf_bias = performance
 #min_perf_pct = 100
 force_latency = 70

[disk]
 elevator = noop

[script]
 script = script.sh

The configuration file consists of several sections and looks very like a Windows ini file.
I’ll not go into detail about the meaning of each parameter, since you can read about it in the man page and the lower part of /etc/sysconfig/sapconf.
Important here is, that some parameters are commented out again. Typically these are performance and energy saving settings. They are always causing debates. Some always want and need the maximum performance whilst other are more serious about low energy consumption. A SAP HANA will have different requirements than other SAP software.
The shipped default is more oriented towards energy saving. If this is not what you need, just remove the comment signs.

As an example how to make changes here, I will increase force_latency to 100.
Again I check the current setting to verify later that it went well.

sles12-sp3:~ # hexdump -e '"%i\n"' /dev/cpu_dma_latency
70

First I copy the tuned.conf in a profile directory of the same name (which I have to create first) in /etc/tuned/:

sles12-sp3:~ # mkdir /etc/tuned/sap-netweaver
sles12-sp3:~ # cp /usr/lib/tuned/sap-netweaver/tuned.conf /etc/tuned/sap-netweaver/

WARNING: Never edit any file beneath /usr/lib/tuned/ directly!
Those files will be overwritten on update. Always create a copy in /etc/tuned// (man 7 tuned-profiles).

Afterwards I change the force_latency value and – this is very important – correct the script path!
As a relative path tuned would search for it in /etc/tuned/sap-netweaver/ and not find it. Since this script applies all the configuration in /etc/sysconfig/sapconf a major part of our configuration would be ignored, if I do nothing here.

WARNING: Do not copy script.sh into the profile directory in /etc/tuned as well! This seems easier, but than it will never updated by a package update and you will miss bug fixes and enhancements.

sles12-sp3:~ # vi /etc/tuned/sap-netweaver/tuned.conf
....
force_latency = 100
....
[script]
script = /usr/lib/tuned/sap-netweaver/script.sh

After a restart of sapconf the new value has been applied by tuned:

sles12-sp3:~ # systemctl restart sapconf.service
sles12-sp3:~ # hexdump -e '"%i\n"' /dev/cpu_dma_latency
100

The log file of sapconf will not show any hint about this change, since it was made by tuned. This can be confusing at first, but remember that sapconf consists of two parts. The tuned configuration (tuned.conf) which is handled by tuned entirely and /etc/sysconfig/sapconf handled by the script referenced in the +[script]+ section, which logs to /var/log/sapconf.

Said that, entries about my last change have to be in the tuned log:

sles12-sp3:~ # tail /var/log/tuned/tuned.log
...
2018-05-15 09:57:10,343 INFO     tuned.profiles.loader: loading profile: sap-netweaver
...
2018-05-15 09:57:10,348 INFO     tuned.daemon.daemon: starting tuning
...
2018-05-15 09:57:10,365 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 100
2018-05-15 09:57:10,366 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/sap-netweaver/script.sh' with argument 'start'

Here we see that:

1. tuned loaded the sap-netweaver profile,
2. force_latency was set to 100 and
3. the sapconf script was called.

NOTE: The name tuned could lead to the impression that this is daemon verifying and correcting all parameters constantly. There is a load based dynamic tuning, but it does not mean that the profile is re-applied regularly. This happens only when a profile is entered!

Let me summarize the most important facts about configuration:

• The configuration is split into two parts:
  • /etc/sysconfig/sapconf
  • /usr/lib/tuned//tuned.conf
• Never edit /usr/lib/tuned//tuned.conf, always make a copy in /etc/tuned/ and don't forget to alter the path to script.sh.
• After making changes, restart sapconf to apply them.
• Check both logfiles /var/log/tuned/tuned.log and /var/log/sapconf.log

That’s it for today.
The next article will cover the update from a previous sapconf.

previous article: sapconf – A way to prepare a SLES system for SAP workload – Part 1

next article: sapconf – A way to prepare a SLES system for SAP workload – Part 3