ServiceLoader do not work in packaged Spring Boot apps

(copy from my GitHub issue: https://github.com/spring-projects/spring-boot/issues/22955)

I noticed that Java's ServiceLoader mechanism doesn't work in packaged Spring Boot apps.

Background

I've tried to use javax.script.ScriptEngineManager which relies on ServiceLoaders. I was able to successfully launch the app from the IDE but not from the command line.

Repro

// build.gradle
plugins {
    id 'java'
    id 'org.springframework.boot' version '2.3.2.RELEASE'
    id 'io.spring.dependency-management' version '1.0.9.RELEASE'
}

tasks.withType(JavaCompile) {
    options.release.set(11) // required Gradle >= 6.6
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.python:jython-slim:2.7.2'
}

// Main.java
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import javax.script.ScriptEngineManager;
import java.util.Objects;

@SpringBootApplication
public class Main {
    public static void main(String[] args) {
        SpringApplication.run(Main.class, args);
        var engine = new ScriptEngineManager().getEngineByName("python");
        Objects.requireNonNull(engine);
        System.out.println("success");
    }
}

It is possible to launch this from the IDE (IntelliJ in my case) but not via the command line:

gradle bootJar && java -jar build/libs/XXX.jar

Temporary workaround

Instead of using the ScriptEngineManager it is possible to directly use Jython's ScriptEngine implementation:

  var engine = new org.python.jsr223.PyScriptEngineFactory().getScriptEngine();

The ServiceLoader mechanism is correctly finding PyScriptEngineFactory. The problem is then a silent failure when it attempts to create a script engine from it. Unfortunately, when you call getEngineByName(String), ScriptEngineManager swallows any exception thrown by getScriptEngine():

try {
ScriptEngine engine = spi.getScriptEngine();
engine.setBindings(getBindings(), ScriptContext.GLOBAL_SCOPE);
return engine;
} catch (Exception exp) {
if (DEBUG) exp.printStackTrace();
}

Your work around doesn't work for me, but that's useful as it allows me to see the exception swallowed by ScriptEngineManager. It is the following:

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.IllegalArgumentException: URI is not hierarchical
at java.io.File.<init>(File.java:418)
at org.python.core.PrePy.getJarFileNameFromURL(PrePy.java:427)
at org.python.core.PrePy._getJarFileName(PrePy.java:362)
at org.python.core.PrePy.getJarFileName(PrePy.java:345)
at org.python.core.PySystemState.doInitialize(PySystemState.java:1195)
at org.python.core.PySystemState.initialize(PySystemState.java:1130)
at org.python.core.PySystemState.initialize(PySystemState.java:1085)
at org.python.core.PySystemState.initialize(PySystemState.java:1080)
at org.python.core.PySystemState.initialize(PySystemState.java:1075)
at org.python.core.PySystemState.initialize(PySystemState.java:1070)
at org.python.core.PySystemState.<init>(PySystemState.java:207)
at org.python.util.PythonInterpreter.threadLocalStateInterpreter(PythonInterpreter.java:80)
at org.python.jsr223.PyScriptEngine.<init>(PyScriptEngine.java:27)
at org.python.jsr223.PyScriptEngineFactory.getScriptEngine(PyScriptEngineFactory.java:85)
at com.example.demo.Gh22955Application.main(Gh22955Application.java:11)
... 8 more

PrePy is making some assumptions about jar:file: URLs that don't hold true in a Spring Boot fat jar. Spring Boot provides an escape hatch for this, allowing a jar file to be automatically unpacked from the fat jar when it's launched. In this case, it's the jython-slim jar that needs to be unpacked. To do that, add the following to your build.gradle:

bootJar {
requiresUnpack "**/jython-slim*"
}