Different encryption Android vs pure Java - RSA/ECB/OAEPWithMD5AndMGF1Padding

I encrypt a string in Android by the public key. However, I get an exception "Decryption error" when I try to decrypt the encrypted string by the private key in pure Java code. Can anyone help to find the problem?

Android code to encrypt

import android.util.Base64;
public static String encryptMessage(final String plainText, final PublicKey publicKey) throws Exception {
    Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithAndMGF1Padding");
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    return Base64.encodeToString(cipher.doFinal(plainText.getBytes()), Base64.NO_WRAP);
}

Pure Java code to decrypt

import java.util.Base64;
public static String decryptMessage(final String encryptedText, final PrivateKey privateKey) throws Exception {
    Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithAndMGF1Padding");
    cipher.init(Cipher.DECRYPT_MODE, privateKey);

    Base64.Decoder decoder = Base64.getDecoder();
    byte[] byteArray = decoder.decode(encryptedText);
    byte[] decryptedArray = cipher.doFinal(byteArray);  // throw exception here
    String plainText = new String(decryptedArray);
    return plainText;
}

You may notice I have to use different Base64 APIs in Android and pure Java. I tried "RSA/ECB/PKCS1Padding", and it can decrypt correctly without the exception. Tried "RSA/ECB/OAEPWithSHA-256AndMGF1Padding" too but got the same exception.

OAEP uses two digests, one for the OAEP label and a second as the basis for MGF1, see RFC 8017, 7.1. RSAES-OAEP, B.1. Hash Functions and B.2. Mask Generation Functions.

The issue is caused because the providers used on both sides of the OP code (Android / API Level 28 and Java 8) apply different MGF1 digests for OAEPWithMD5AndMGF1Padding.

On both sides the relevant parameters (provider, OAEP digest, MGF, MGF1 digest) can be determined after the initialization of the cipher e.g. with:

OAEPParameterSpec parameterSpec = cipher.getParameters().getParameterSpec(OAEPParameterSpec.class);
System.out.println("Provider: " + cipher.getProvider().getName());          
System.out.println("OAEP digest: " + parameterSpec.getDigestAlgorithm());  
System.out.println("OAEP MGF : " + parameterSpec.getMGFAlgorithm());        
System.out.println("OAEP MGF1 digest: " + ((MGF1ParameterSpec)parameterSpec.getMGFParameters()).getDigestAlgorithm()); 

With this, MD5 is determined as MGF1 digest on the Android side and SHA-1 on the Java side. On both sides MD5 is used as OAEP digest. The issue can be fixed if the digests are explicitly set with OAEPParameterSpec so that the same digests are used on both sides.

For example, the following code on the Java side ensures that MD5 is used as OAEP and MGF1 digest, analogous to the Android side.

OAEPParameterSpec oaepParameterSpecDec = new OAEPParameterSpec("MD5", "MGF1", new MGF1ParameterSpec("MD5"), PSource.PSpecified.DEFAULT);
cipher.init(Cipher.DECRYPT_MODE, privateKey, oaepParameterSpecDec);

The same applies to OAEPWithSHA-256AndMGF1Padding.

Note also that RFC 8017 in B.1. Hash Functions recommends SHA-1 and SHA-2 for RSAES-OAEP, but not MD5.