kiwi image build fails with KiwiDecodingError

SUSE Linux Enterprise Server 15 All versions

At the time of writing this document it in a specific locale setup is observed that kiwi build operations fail like in this scenario :
# kiwi-ng --debug --type docker system build --description . --allow-existing-root --target-dir /tmp/
[...]
[ DEBUG   ]: 13:17:42 | EXEC: [rpm --root /tmp/build/image-root -qa --qf %{NAME}|\n --changelog --dbpath /usr/lib/sysimage/rpm]
[ WARNING ]: 13:17:43 | Failed decoding literal. Forcing UTF-8 decoding
[ ERROR   ]: 13:17:43 | KiwiDecodingError: Locale setup is not utf-8 compatible

The problem is observed in a setup installed with en_US.UTF-8 locale :
# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

However the user has changed the LC_* settings to "C" :
# locale
LANG=en_US.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

In the shell invoking the kiwi-ng command set the environment variable :
LC_ALL=C.UTF-8

Setting LC_ALL to C results in LC_CTYPE seen by the externally run rpm command being C rather than C.UTF-8 and causes running rpm (to get --changelog of our packages) to generate output that includes extended ASCII characters that are not locale text and are not UTF-8. Which is the specific failure observed. To prevent this, the correct setting to use for LC_ALL is not C it is C.UTF-8. They are not equivalent in this context.

Is this expected behavior ? Yes and the cause has nothing to do with the kiwi configuration XML locale setting. rpm has no awareness of kiwi configuration and will only generate output supported by python (kiwi) if host configuration (locale environment) indicates to output UTF-8.

You can set everything except LC_ALL to C before running a python program and the LC_CTYPE seen by an external program to python that's run by python SubProcess will still be C.UTF-8. NB: That includes if you set LC_CTYPE itself to C before starting the python program then (as long as LC_ALL is empty) LC_CTYPE will still be C.UTF-8 when seen by the external program.

Setting LC_ALL to C.UTF-8 instead of C prevents the failure, but achieves the goal of the locale being C.
In other words, wanting to change the locale via environment variables that include LC_ALL when using python programs like kiwi should not be limited to LC_ALL=<locale name>. They should instead be expressed as the UTF-8 version of that locale, e.g. LC_ALL=C.UTF-8.