Apache Pulsar unable to validate issuer certificate when attempting to load OpenID Connect provider

I've attempted to follow the instructions on https://pulsar.apache.org/docs/3.0.x/security-openid-connect/ in order to enable OpenID Connect Authentication, but can't seem to get passed a certificate error.

I believe this error stems from pulsar's AuthenticationProviderOpenID.java class, where in the initialise method it attempts to setup the SSL context to enable a secure connection to the issuer url provided.

In the example broker.conf it says that the "openIDTokenIssuerTrustCertsFilePath" is an optional setting and that it will use the trust store of the JVM. So I ensured that the required certificates where added to the trust store found at ".../jre/lib/security/cacerts" on my broker instance and left that config value as blank. This did not work, I was presented with the error:

Caused by: java.lang.IllegalArgumentException: File does not contain valid certificates:
....
Caused by: java.security.cert.CertificateException: could not find certificate file:
at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:68) ~[io.netty-netty-handler-4.1.89.Final.jar:4.1.89.Final]

This to me seemed to indicate that it was expecting a .pem file to be supplied, so I copied the pem file onto the broker server and supplied the absolute path to the file as openIDTokenIssuerTrustCertsFilePath=/opt/pulsar/certificates/certificate.pem

but I received the error

Caused by: java.security.cert.CertificateException: found no certificates in input stream

Here is a example of the broker.conf settings I'm using.

broker.conf settings
authenticationProviders=org.apache.pulsar.broker.authentication.oidc.AuthenticationProviderOpenID

openIDAllowedTokenIssuers=https://login-stg.somewebsite.com/
openIDAllowedAudiences=https://auth0-some-website.com/maas, https://some-stg.some-dev.auth0.com/userinfo
        
openIDTokenIssuerTrustCertsFilePath=
openIDRoleClaim=https://some-web.com/role

> In the example broker.conf it says that the
> "openIDTokenIssuerTrustCertsFilePath" is an optional setting and that
> it will use the trust store of the JVM. So I ensured that the required
> certificates where added to the trust store found at
> ".../jre/lib/security/cacerts" on my broker instance and left that
> config value as blank.

It will work correctly when you remove the openIDTokenIssuerTrustCertsFilePath= from your broker.conf file. Having a blank openIDTokenIssuerTrustCertsFilePath= is incorrectly configuring the provider to attempt to load the "" file, which explains the unhelpful error message.

Note: https://github.com/apache/pulsar/pull/20745 should improve the behavior in a future release of pulsar by interpreting an empty configuration as using the OS's trust store.

> Caused by: java.security.cert.CertificateException: found no certificates in input stream

In my testing, I observed this same error when the file was not in PEM format. I verified that the feature does work for custom trust stores configured using correctly formated files with these modified tests: https://github.com/apache/pulsar/pull/20746.

Here is my full stack trace for the incorrectly formatted ca cert:

java.lang.IllegalArgumentException: File does not contain valid certificates: /Users/michaelmarshall/dev/apache/pulsar/pulsar-broker-auth-oidc/target/test-classes/certificate-authority/jks/broker.keystore.jks

	at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:261)
	at org.apache.pulsar.broker.authentication.oidc.AuthenticationProviderOpenID.initialize(AuthenticationProviderOpenID.java:172)
	at org.apache.pulsar.broker.authentication.oidc.AuthenticationProviderOpenIDIntegrationTest.testKidCacheMissWhenRefreshConfigZero(AuthenticationProviderOpenIDIntegrationTest.java:347)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
Caused by: java.security.cert.CertificateException: found no certificates in input stream
	at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:107)
	at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:63)
	at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1219)
	at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:259)
	... 29 more