This article has been contributed by Jose Gomez, Full Stack Web Developer at SUSE.   

Attention: This article does not intend to provide a guide for the Open Build Service (OBS). If you want to learn more about OBS, visit the project’s Web page at http://openbuildservice.org/ and read the specific documentation there http://openbuildservice.org/help/.

Background

The Open Build Service (OBS) is a generic system to build and distribute packages from sources in an automatic, consistent and reproducible way. It makes it possible to release software for a wide range of operating systems and hardware architectures. While there exists only one Open Build Service, it maintains two separate collections, or APIs, of repositories:

• The external Build Service at https://openbuildservice.org/ can be used by everyone. When you log in to this Web site, you can view all projects available in the external build service.
• The Internal Build Service (IBS) is the instance where the SUSE products are built and maintained. Only SUSE employees can access this Web site using their personal log in credentials.

As of Sept 1st, 2022, the IBS from SUSE is using multi-factor authentication (MFA). It follows the following RFC’s:

• RFC7235: Hypertext Transport Protocol (HTTP/1.1): Authentication
• Signing HTTP Messages, draft-cavage-http-signatures-12
• Signing HTTP Messages, draft-richanna-http-message-signatures-00

RFC7235 section 4.1 defines how an HTTP Server requests authentication from a client. The server responds with an HTTP status 401 (‘Unauthorized’), and the header WWW-Authenticate provides instructions to the client on how to provide authentication back to the server.

Quoting section 4.1 of RFC7235:

WWW-Authenticate: Newauth realm="apps", type=1, title="Login to \"apps\"", Basic realm="simple"

This header field contains two challenges; one for the “Newauth” scheme with a realm value of “apps”, and two additional parameters “type” and “title”, and another one for the “Basic” scheme with a realm value of “simple”.

Note: The challenge grammar production uses the list syntax as well.
Therefore, a sequence of comma, white space, and comma can be considered either as applying to the preceding challenge, or to be an empty entry in the list of challenges.
In practice, this ambiguity does not affect the semantics of the header field value and thus is harmless.

The Build Service follows this very same convention, although the challenge is the one defined in Signing HTTP Messages, draft-cavage-http-signatures-12 and the signature algorithm is defined in Signing HTTP Messages, draft-richanna-http-message-signatures-00.

The procedure is practically the same, including the signature algorithm used (referred as “ssh” by the Build Service).

hs2019 signature algorithm is not completely standardized but for our purposes we’ll define it as a hash + sign algorithm that varies depending on the private key involved.

The implementation used in this flow is openssh‘s implementation which automatically selects the most secure hash & signing algorithm possible.

Our experiments while writing this article showed that:

• RSA 4096 keys it uses: sha512 for hashing and rsa-sha2-512 for signing.
• ED25519 keys it uses: sha512 for hashing and ssh-ed25519 for signing.

For simplicity, we’ll refer to this as SSHSIG in this article.

Authentication flow

In this section, we will describe each single authentication step. At the end of each step you will find a recap of the process. The recap is frozen in timestamp 1664187470 to ensure that the full signing procedure is correct and reproduceable.

Below is the key pair used to generate the signatures in this document: it is not registered in any suse.de service as it was created with the sole purpose of validating the output of the commands below. It was generated with:

ssh-keygen -t ed25519 -C sample-mfa-flow@ibs -f sample-mfa-flow.key

Private Key:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACBIqlwQI+bxWOj1TOdlL4z9AuOZdeYvOgHGQhtnxn+g+QAAAJiRS1EekUtR
HgAAAAtzc2gtZWQyNTUxOQAAACBIqlwQI+bxWOj1TOdlL4z9AuOZdeYvOgHGQhtnxn+g+Q
AAAECrZDKH46WiRLiazilOn4+BlnESdV8CNReMvlm2Pr6Yr0iqXBAj5vFY6PVM52UvjP0C
45l15i86AcZCG2fGf6D5AAAAE3NhbXBsZS1tZmEtZmxvd0BpYnMBAg==
-----END OPENSSH PRIVATE KEY-----

1. Access a protected resource (Challenge) ❌✋

We’ve chosen /about as our probe endpoint to get authentication instructions.

$ curl -s -I https://build-service.suse.internal/about
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... we don't care about the rest...

WWW-Authenticate tells us the authentication challenge the client must fulfill. In this case, Build Service tells us we need to use the Signature Authentication method (defined in Signing HTTP Messages, draft-cavage-http-signatures-12) and parameters for the signature to be built.

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"

2. Understanding the Authentication Challenge (Parse Challenge) 🤓📄

Let’s parse the challenge:

Signature realm="Use your developer account",headers="(created)"

This is interpreted as:

Challenge Type: Signature
Challenge Parameters:
  realm = "Use your developer account"
  payload = "(created)"

Section 2.3 of Signing HTTP Messages, draft-cavage-http-signatures-12 specifies what are the minimum fields to use in a challenge. Headers surrounded by parentheses () imply calculated fields, like (created) which is the UNIX timestamp of the request.

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... we don't care about the rest...

2. Parse Challenge:

Challenge Type: Signature
Challenge Parameters:
  realm = "Use your developer account"
  payload = "(created)"

3. Building the Signature String (Payload) 👷📄

Following Section 2.3 of Signing HTTP Messages, draft-cavage-http-signatures-12, it states (in short) that the requested headers must be written in the following format:

header: value
x-empty-header: 
(calculated-header): calculated-value

Following the example response from the Build Service, we have:

Signature realm="Use your developer account",headers="(created)"

It requires only (created) header to be included in the signature string. That translates then to the following signature string:

(created): 1664187470

This signature string now must be signed with a private SSH Key.

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... we don't care about the rest...

2. Parse Challenge:

Challenge Type: Signature
Challenge Parameters:
  realm = "Use your developer account"
  payload = "(created)"

3. Build Signature String (Payload):

(created): 1664187470

4. Signing/hashing the Payload ✍📄

This is the most important part of the flow. We will sign/hash our signature string with the SSH private key (asymmetric crypto 101, you SIGN with your PrivKey, you VERIFY with the PubKey)

The Build Service’s signing algorithm is provided by openssh with ssh-keygen -Y sign (available in OpenSSH >= 8.0).

Build Service expects the signatures to be the output of the following command:

$ ssh-keygen -Y sign -f "%private key%" -q -n "%realm%" < <(echo -n "%signature string%")

Following our example with the build service, it’ll translate to the command below:

$ CREATED_TIMESTAMP=1664187470 # this will make sense in a second
$ PRIVATE_KEY_PATH="%path to private key%"
$ SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"
$ ssh-keygen -Y sign -f $PRIVATE_KEY_PATH -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING")

The output of ssh-keygen -Y sign command is an Armored SSH Signature defined in OpenSSH PROTOCOL.sshsig, which is a Base64 representation of the SSH Signature Buffer. It looks like the following:

-----BEGIN SSH SIGNATURE-----
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmX
XmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNo
YTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGR
J3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==
-----END SSH SIGNATURE-----

Note: You can replicate this signature with the private key given in the intro above. However, the resulting signature length will vary on your type of SSH key. For example, RSA keys produce a significantly larger output than ED25519 keys.

To be able to pass this in the Authentication header, we just need to strip the start/end delimiters & strip newlines (as an HTTP header cannot contain newlines).

After removing the delimiters and stripping newlines, we have:

$ CREATED_TIMESTAMP=1664187470 # this will make sense in a second, I promise, really, wait for it
$ PRIVATE_KEY_PATH="%path to private key%"
$ SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"
$ ssh-keygen -Y sign -f $PRIVATE_KEY_PATH -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n"

U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==

With this, now we can build the Authentication Header.

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401
www-authenticate: Signature realm="Use your developer account",headers="(created)"

2. Parse Challenge:

Challenge Type: Signature
Challenge Parameters:
  realm = "Use your developer account"
  payload = "(created)"

3. Build Signature String:

(created): 1664187470

4. Signing the Payload (Payload) (we’ll use the stripped version right away):

$ CREATED_TIMESTAMP=1664187470 # this will make sense in a second, I promise, really, wait for it, be patient
$ PRIVATE_KEY_PATH="%path to private key%"
$ SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"
$ ssh-keygen -Y sign -f $PRIVATE_KEY_PATH -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n"
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==

5. Building the Authentication header 👷🔑

We now provide the result of the signature to the server and provide hints on how to verify this signature. As per Signing HTTP Messages, draft-cavage-http-signatures-12, section 3.1.2, the Authorization header looks like:

Authorization: Signature keyId="rsa-key-1",algorithm="HS2019",headers="(request-target) (created) host digest content-length",signature="Base64(RSA-SHA512(signature string))"

It provides hints for the server to be able to verify our signature process on their side.

• keyId = a shared identifier between the server & us to find the key that signed this.
• algorithm = the algorithm used to make the signature.
• headers = headers involved in the signature string.
• signature = the final signature result.

For the Build Service implementation there are some minor differences:

• algorithm="HS2019" is replaced by algorithm="ssh".
• signature=Base64(RSA-SHA512(signature string)) is replaced by signature="Base64(SSHSIG(private-key, signature-string))".

Authorization: Signature keyId="%build-service-username%",algorithm="ssh",headers="(created)",signature="Base64(SSHSIG(private-key, signature-string)),created=%timestamp%"

• keyId = The IBS Username
• algorithm=ssh. This is a constant, unique to the build service.
• headers = headers involved in the signature string, those are the same the Build Service provides on step 1.
• signature = the final signature result.
• created = unix timestamp used in the signature string. It must match the signature string one. (This is the reason why it was always separated in the code samples)

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... we don't care about the rest...

2. Parse Challenge:

Challenge Type: Signature
Challenge Parameters
  realm = "Use your developer account"
  payload = "(created)"

3. Build Signature String:

(created): 1664187470

4. Signing the Payload (Payload) (we’ll use the stripped version right away):

$ CREATED_TIMESTAMP=1664187470 # this will make sense in a second, I promise, really, wait for it, be patient, we're almost there
$ PRIVATE_KEY_PATH="%path to private key%"
$ SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"
$ ssh-keygen -Y sign -f $PRIVATE_KEY_PATH -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n"
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==

5. Build the Authorization header

Authorization: Signature keyId="%ibs-username%",algorithm="ssh",signature="%username",headers="(created)",created="%created unix timestamp%'

For example:

Authorization: Signature keyId="dummy-username",algorithm="ssh",signature="U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==",headers="(created)",created="$CREATED_TIMESTAMP" # see? that's why we were separating the timestamp.

6. Retry the request 🔄

With the header built, we can retry the request this time providing the Authorization header we’ve just built.

Here is a small bash snippet to show the result in the console (removed the leading $ for a fast copy-paste):

Note: Remember that the private key used in this document is not registered anywhere, attempting to use it will incur in authentication errors.

PRIVATE_KEY_PATH="%path to private key%"
IBS_USERNAME=dummy-username # replace this with the username matching the ssh key in privacy idea.
CREATED_TIMESTAMP=$(date +%s)
SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"

SIGNATURE=$(ssh-keygen -Y sign -f "$PRIVATE_KEY_PATH" -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n")
curl -I -H 'Authorization: Signature keyId="'$IBS_USERNAME'",algorithm="ssh",signature="'$SIGNATURE'",headers="(created)",created="'$CREATED_TIMESTAMP'"' https://api.suse.de/about

After this, there are only two outcomes:

1. Unsuccessful challenge (failed auth): The Build Service will return a HTTP 401 indicating that there was a problem with the authentication (and no more details are given).

# example curl for failed auth response
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... snip snip snip ... #

2. Success: The Build Service will process the request and append a Set-Cookie header. This cookie will allow us not to redo the challenge every time at the expense of the HTTP clients being stateful.

# example curl for successful auth response

HTTP/2 200 
# ... snip snip snip ... #
set-cookie: openSUSE_session=XXXXXXXXXXXXXXXX; path=/; Max-Age=86400; Secure; HttpOnly; domain=.suse.de
# ... snip snip snip ... #

Recap

1. Challenge:

$ curl -s -I https://build-service.suse.internal/about | egrep -i "(^HTTP|www-authenticate)"
HTTP/2 401 
www-authenticate: Signature realm="Use your developer account",headers="(created)"
# ... we don't care about the rest...

2. Parse Challenge:

Challenge Type: Signature
Challenge Parameters
  realm = "Use your developer account"
  payload = "(created)"

3. Build Signature String:

(created): 1664187470

4. Signing the Payload (Payload) (we’ll use the stripped version right away):

$ CREATED_TIMESTAMP=1664187470 # this will make sense in a second, I promise, really, wait for it, be patient
$ PRIVATE_KEY_PATH="%path to private key%"
$ SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"
$ ssh-keygen -Y sign -f $PRIVATE_KEY_PATH -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n"
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==

5. Build the Authorization header

Authorization: Signature keyId="%ibs-username%",algorithm="ssh",signature="%username",headers="(created)",created="%created unix timestamp%'

For example:

Authorization: Signature keyId="dummy-username",algorithm="ssh",signature="U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==",headers="(created)",created="1664187470"

6. Retry the request

# this request will fail, is just for demonstrative purposes.

curl -I -H 'Authorization: Signature keyId="dummy-username",algorithm="ssh",signature="U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgSKpcECPm8Vjo9UznZS+M/QLjmXXmLzoBxkIbZ8Z/oPkAAAAaVXNlIHlvdXIgZGV2ZWxvcGVyIGFjY291bnQAAAAAAAAABnNoYTUxMgAAAFMAAAALc3NoLWVkMjU1MTkAAABA8cmvTy1PgpW2XhHWxQ1yw/wPGAfT2M3CGRJ3II7uT5Orqn1a0bWlo/lEV0WiqP+pPcQdajQ4a2YGJvpfzT1uBA==",headers="(created)",created="1664187470"' https://api.suse.de/about

A working example in bash (removed the leading $ for a fast copy-paste):

PRIVATE_KEY_PATH="%path to private key%"
IBS_USERNAME=dummy-username # replace this with the username matching the ssh key in privacy idea.
CREATED_TIMESTAMP=$(date +%s)
SIGNATURE_STRING="(created): $CREATED_TIMESTAMP"

SIGNATURE=$(ssh-keygen -Y sign -f "$PRIVATE_KEY_PATH" -q -n "Use your developer account" < <(echo -n "$SIGNATURE_STRING") | tail -n +2 | head -n -1 | tr -d "\n")
curl -I -H 'Authorization: Signature keyId="'$IBS_USERNAME'",algorithm="ssh",signature="'$SIGNATURE'",headers="(created)",created="'$CREATED_TIMESTAMP'"' https://api.suse.de/about

Summary

As mentioned before, in this article we outlined a general concept for MFA on the Build Service, based on our experiments with the internal instance. If you want to try something similar for the external Open Build Service instance, we would be happy if you’d share your feedback about your findings. And now, all that remains for us to say is: have a lot of fun!