Google KSM does not sign RSA PKCS v1.5 through OpenSSL / PKCS#11 with "mechanism 0xd is not permitted for key"

We are trying to setup a mTLS connection using a private key that is generated and stored in a HSM. We are using Google KMS for the HSM part.

Our application is a Nodejs app that uses axios to make requests, and we're configuring the underlying openssl / pkcs#11 interface as follows:

httpsAgent = new Agent({
  privateKeyEngine: 'pkcs11',
  privateKeyIdentifier,
  cert: fullCert
})

This worked fine in our development setup, where we had a LetsEncrypt Certificate using 2048 bit RSA key using RSA-PSS as the signature algorithm. We imported the key we got from certbot into the HSM, choosing rsa-sign-pss-2048-sha256 as its signing algorithm and with a key purpose of ASYMMETRIC_SIGNING.

Now in production however, we are required to use a cert from specific CAs (don't ask). We generated a prod key in the HSM directly (so we are unable to extract the private key), used OpenSSL to generate a CSR for the CA but noticed that they do not accept CSR using RSA-PSS.

We had to downgrade the signing algorithm to PKCS v1.5 so that the CA would accept the CSR.

So the CA accepted the CSR, signed the certificate and we set everything up identically to our dev setup: However, we now get the following error when trying to establish a connection:

module:pkcs11_private_encrypt:Mechanism invalid:p11_rsa.c:116

Additionally, the KMS integration by Google logs the following:

returning 0x68 from C_SignInit due to status FAILED_PRECONDITION: at preconditions.cc:47: mechanism 0xd is not permitted for key
returning 0x70 from C_SignInit due to status INVALID_ARGUMENT: at crypter_ops.cc:62: mechanism 0x3 is not valid for operation sign

Now, I'm not really an expert in cryptography, but the way I understood it was:

• If I use "2048 bit RSA key PKCS#1 v1.5 padding - SHA256 Digest" as the algorithm for the key in Google KMS, and I'm successfully able to generate a CSR using OpenSSL, shouldn't this work the same compared to using "2048 bit RSA key PSS Padding - SHA256 Digest"?

We're using Node v16, Debian 11.6 with OpenSSL 1.1.1n. I've tried to upgrade the PKCS#11 engine to the latest version as well but to no effect.

Is there any flag or config I need to set so that Node/OpenSSL know which mechanism to use? OpenSSL reports that RSA_PKCS is supported by the engine.

So, for anyone trying the same thing. I managed to find the issue using the PKCS#11 Spy included in opensc:

30: C_SignInit
[in] hSession = 0xff0ba74ce08430ad
pMechanism->type=CKM_RSA_PSS

I noticed that it somehow selected the wrong signing algorithm when trying to sign. It used CKM_RSA_PSS instead of CKM_RSA_PKCS, but this isn't what the configured key is capable of. I'm not sure why it does that, maybe it was simply a configuration issue and RSA_PSS was the selected default, which made it work by chance in our test environment.

I managed to force the correct behavior by adding some flags to the https.Agent in node, namely sigalgs and setting a the max TLS version to 1.2:

httpsAgent = new Agent({
  privateKeyEngine: 'pkcs11',
  privateKeyIdentifier,
  cert,

  maxVersion: 'TLSv1.2', // use TLS v1.2 max
  sigalgs:'RSA+SHA256' // choose the right signing algorithm
})

This led to OpenSSL using the correct signing algorithm. Setting the TLS version to v1.2 was necessary as well, but it was an unrelated follow-up problem (I included this for completion if someone has a similar setup going on).