JLink does not produce redistributable image

For a while now I've been working with modular projects, but due to being constrained with filename and automatic modules, I had never got a chance to work with jlink tool to produce a redistributable application image. Today I've opted to start an independent project which does not import any external dependencies to prevent usage of the compatibility mode. The project consists of 3 modules and is in maven, so I will only be posting the jlink command snippet I'm using.

Project for reference: https://gitlab.com/Dragas/edu-day-demo, checkout the modules-full tag. Project is built with package goal, to prevent polluting your local .m2 repository. Project is already configured to pull dependencies so packaging and deployment would be easier.

The command I used to generate the jlinked image was as follows:

jlink \
   --module-path edu-day-runtime/target/dependency/:edu-day-runtime/target/ \
   --add-modules ALL-MODULE-PATH \
   --output edu-day-jlinked \
   --launcher edurun=edu.day.runtime

Invoking the command does indeed generate a jlinked image, which contains minimum required modules, java libraries and JVM binaries to run the project. Invoking the machine that built the image

edu-day-jlinked/bin/edurun 1 1

does run the project and outputs the following

Result of sum is 2

Meanwhile, attempting to run the same in containerized environment (here i'm using bash:5, a non-java image to simulate an environment where java is not installed) does not yield similar results. Instead, the shell does not seem to find a binary named java

docker run -it -v "$(pwd)/edu-day-jlinked:/app" bash:5
...(in container)
bash-5.0# /app/bin/edurun 1 1
/app/bin/edurun: line 4: /app/bin/java: not found

Upon inspection, the folder does indeed contain the binary called java

bash-5.0# ls -la
total 52
drwxr-xr-x    2 1000     1000          4096 Aug 23 07:53 .
drwxr-xr-x    7 1000     1000          4096 Aug 23 07:53 ..
-rwxr-xr-x    1 1000     1000           116 Aug 23 07:53 edurun
-rwxr-xr-x    1 1000     1000         16688 Aug 23 07:53 java
-rwxr-xr-x    1 1000     1000         16712 Aug 23 07:53 keytool

But even invoking it directly (to show the help message) does not yield any results, besides the same message that the binary cannot be found

(in /app/bin/ folder)
bash-5.0# ./java 
bash: ./java: No such file or directory

What is more interesting is that even the keytool binary returns the same error

(in /app/bin/ folder)
bash-5.0# ./keytool
bash: ./keytool: No such file or directory

This raises a question: what went wrong? I haven't yet delved deeper into how jlink works, but my speculation is that it copies the binaries from my own java installation (openjdk 11.0.8+10 from arch repositories), and considers that to be redistributable. Or did I just miss some command line options?

Your issue is that the test container (bash:5) doesn't use the same version of the run-time linker as the java environment.

The binary produced by the jlink will only run if there is a compatible linux run-time linker on the system.

The purpose of the run-time linker is to configure the binary for execution on the system - at the time you are building an executable the default run-time linker is hard-coded into the binary. You can inspect the run-time linker using a tool such as readelf -l, or ldd (ldd only works if it can find the run-time linker)

The default run-time linker for amd64 linux (e.g. ubuntu) is: /lib64/ld-linux-x86-64.so.2

The default run-time linker for i386 linux is: /lib/ld-linux.so.2

On a bash:5 container, the default run-time linker is: /lib/ld-musl-x86_64.so.1

This is not compatible with the run-time linker for the jdk

The error: /app/bin/java: not found is caused because the run-time linker cannot be found for the binary. A dirty test of a jlinked VM in a bash:5 container gives the same error.

When I get the run-time linker for the java I've used:

$ docker run --rm -it -v (pwd)/edu-day-jlinked64:/app -w /here bash:5 bash

bash-5.0# /app/bin/java
bash: /app/bin/java: No such file or directory
bash-5.0# strings -a /app/bin/java | grep '^/lib'
/lib64/ld-linux-x86-64.so.2
bash-5.0# ls -l /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory

Testing with the run-time linker that's on-board:

bash-5.0# /lib/ld-musl-x86_64.so.1 --list /app/bin/java
/lib64/ld-linux-x86-64.so.2 (0x7fe2852a3000)
libjli.so => /app/bin/../lib/libjli.so (0x7fe28528c000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fe2852a3000)
libz.so.1 => /lib/libz.so.1 (0x7fe285272000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7fe2852a3000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fe2852a3000)
Error relocating /app/bin/../lib/libjli.so: __snprintf_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __vfprintf_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __read_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __memmove_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __printf_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __fprintf_chk: symbol not found
Error relocating /app/bin/../lib/libjli.so: __sprintf_chk: symbol not found

so it definitely won't work here.

Let's use something 'standard'. As I had built the jlinked app in an ubuntu:focal container, with an installed version of java let's use one that doesn't have java built-in:

$ docker run --rm -it -v $(pwd)/edu-day-jlinked64:/app -w /here ubuntu:focal bash
root@865c9c12c029:/here# /app/bin/java
Usage: java [options] <mainclass> [args...]
           (to execute a class)
   or  java [options] -jar <jarfile> [args...]
           (to execute a jar file)
   or  java [options] -m <module>[/<mainclass>] [args...]
       java [options] --module <module>[/<mainclass>] [args...]
           (to execute the main class in a module)
   or  java [options] <sourcefile> [args]
           (to execute a single source-file program)

so it will work in this case.

Reproducibility:

Built using:

$ docker run --rm -it -v $(pwd):/here -w /here ubuntu:focal bash

# apt-get update
# DEBIAN_FRONTEND=noninteractive apt-get install -y git openjdk-14-jdk maven
# git clone https://gitlab.com/Dragas/edu-day-demo .
# git checkout modules-full
# ./mvnw package
# rm -rf edu-day-runtime/target/classes
# jlink --module-path edu-day-runtime/target/dependency/:edu-day-runtime/target/ --add-modules ALL-MODULE-PATH --output edu-day-jlinked    --launcher edurun=edu.day.runtime
# ./edu-day-jlinked/bin/edurun 1 1
Result of sum is 2

In an adjacent directory:

$ docker run --rm -it -v $(pwd)/edu-day-jlinked:/app -w /here bash:5 bash
bash-5.0# /app/bin/edurun 1 1
/app/bin/edurun: line 4: /app/bin/java: not found

In another directory:

$ docker run --rm -it -v $(pwd)/edu-day-jlinked:/app -w /here ubuntu:focal bash
root@200b4a98f9ee:/here# /app/bin/edurun 1 1
Result of sum is 2

TL;DR — The bash:5 image uses a C library that is binary incompatible with the C library that was linked with your edu-day-jlinked/bin/java executable.

The long version

> „…This raises a question: what went wrong?…“

What's going wrong is your app/bin/java binary is failing to find the C library that it was originally linked to when you built your edu-day-jlinked executable on whatever machine you built it on locally.

The problem occurs because the java binary that jlink produced is linked to the GNU glibc library that your locally-installed JDK uses…

$ ldd edu-day-demo-modules-full/edu-day-jlinked/bin/java
        …
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa61a95b000)
        …

Whereas, the bash:5 image runs in the Busybox Linux distribution. And Busybox does not use glibc…

bash-5.0# ldd app/bin/java
        …
        libjli.so => app/bin/../lib/jli/libjli.so (0x7f572a16d000)
        …
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f572a19f000)
Error relocating app/bin/../lib/jli/libjli.so: __snprintf_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __vfprintf_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __read_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __memmove_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __printf_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __fprintf_chk: symbol not found
Error relocating app/bin/../lib/jli/libjli.so: __sprintf_chk: symbol not found

It uses a different C library: musl…

bash-5.0# find / -name '*musl*'
/lib/libc.musl-x86_64.so.1
/lib/ld-musl-x86_64.so.1

It helps to understand the Linking process. And it also helps to bear in mind that JLink builds a custom executable for a specific environment.

Your trial run on your local machine worked, because jlink built the executable specifically for your local environment.

The proposed solution

> „…docker here is intended to simulate an environment which does not have java installed…“

Here is a Dockerfile that successfully builds your application and the resulting image „does not have java installed“…

FROM maven:3.6.1-jdk-13-alpine as build

WORKDIR /app

COPY pom.xml .
COPY edu-day-sum edu-day-sum
COPY edu-day-runtime edu-day-runtime
COPY edu-day-api edu-day-api

RUN mvn package && \
   --module-path ${JAVA_HOME}/jmods:edu-day-runtime/target/dependency/:edu-day-runtime/target/edu-day-runtime-1.0-SNAPSHOT.jar \
   --add-modules ALL-MODULE-PATH \
   --output edu-day-jlinked \
   --launcher edurun=edu.day.runtime

FROM alpine:latest

COPY --from=build /app/edu-day-jlinked /app

ENTRYPOINT ["/app/bin/edurun"]
CMD ["1", "1"]

Docker best practice advises: „Use multi-stage builds“ (like in the above Dockerfile) when your aim is to build „an environment which does not have java installed“.

The FROM maven:3.6.1-jdk-13-alpine stage of the multi-stage build, uses an alpine Linux image that has both Maven and a JDK specifically built to be compatible with the alpine distribution.

The FROM alpine:latest is a very small linux distro that does not have Java on it. The maven:3.6.1-jdk-13-alpine layer is discarded as the Docker best practice docs says. The only java in the resulting image is the one in app/bin.