Error in Azure App Service: java.lang.UnsatisfiedLinkError: failed to load the required native library

I have a Spring Boot App which I want to deploy to Azure App Service as a war.
The spring boot app has REST APIs. And the App Service is configured as below:

• OS: Linux
• Publish: Code
• Runtime Stack: Tomcat 9

My pom.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
		 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>sdk-hlf</groupId>
	<artifactId>sdk-hlf-1.4.4</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>war</packaging>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.3.0.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
		<tomcat.version>9.0.20</tomcat.version>
		<start-class>com.example.app.MainClass</start-class> <!-- edited for security reasons -->
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-api</artifactId>
		</dependency>

		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.12</version>
			<scope>provided</scope>
		</dependency>

		<dependency>
			<groupId>org.hyperledger.fabric</groupId>
			<artifactId>fabric-gateway-java</artifactId>
			<version>1.4.4</version>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-tomcat</artifactId>
			<scope>provided</scope>
		</dependency>
	</dependencies>

	<build>
		<finalName>demo</finalName>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<executions>
					<execution>
						<id>repackage</id>
						<goals>
							<goal>repackage</goal>
						</goals>
						<configuration>
							<finalName>demo</finalName>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

When I run the App in my laptop, it works fine.
But the problem occurs after deploying the app to Azure. It is deployed successfully, but when I hit an API of my app I get the below error:

{
    "timestamp": "2020-09-12T15:06:06.732+00:00",
    "status": 500,
    "error": "Internal Server Error",
    "message": "",
    "path": "/api/get" //edited for security reasons
}

Below is a portion of the logs:

2020-09-12T15:35:31.508024047Z: [INFO]
2020-09-12T15:35:31.508053247Z: [INFO]  java.lang.UnsatisfiedLinkError: failed to load the required native library
2020-09-12T15:35:31.508057547Z: [INFO]          at io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:474) ~[netty-handler-4.1.49
.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508080348Z: [INFO]          at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContex
t.java:196) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508084948Z: [INFO]          at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContex
t.java:185) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508089148Z: [INFO]          at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:34) ~[netty-handler-4.1.4
9.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508093148Z: [INFO]          at io.netty.handler.ssl.OpenSslClientContext.<init>(OpenSslClientContext.java:189) ~[netty-
handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508097248Z: [INFO]          at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:827) ~[netty-ha
ndler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508101248Z: [INFO]          at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:576) ~[netty-handler
-4.1.49.Final.jar:4.1.49.Final]

I dont know whats the problem here?

I found the issue, 'fabric gateway' depends on 'boring ssl', and 'boring ssl' depends on a DLL file in case on Windows and an SO file in case of Linux. In my dependencies list, I see both SO and DLL files packed in 'netty-tcnative-boringssl-static-2.0.26.Final.jar\META-INF\native' folder. This folder has following files.

> libnetty_tcnative_linux_x86_64.so
> libnetty_tcnative_osx_x86_64.jnilib
> netty_tcnative_windows_x86_64.dll

JVM is supposed to pick the necessary native file (DLL or SO file based on host OS) from META-INF/native folder, but picks only in case of Windows, but not linux. So, for this, I've picked up the SO file from jar, put it in a folder on Linux box and 'loaded' the SO file using System.load as follows. This has resolved the issue.

static{
	try{
		String path="/apps/config/libnetty_tcnative_linux_x86_64.so";
		Path path = Paths.get(path);

		if (Files.exists(path)) {
			System.load(path);
			System.out.println("Native library loaded successfully");
		}
	}catch (Throwable e){
		System.out.println("Native library load failed");
		e.printStackTrace();
	}
}

Static blocks load first when the class is being loaded, so, I've included above static block in the 'entry point' Java class, which ensures that SO file is loaded before any Java class is loaded.

Now, the 'boring ssl jar' that should be included is based on your host OS, read the following documentation on how you can specify stuff in your pom.xml for appropriate JAR to be included. If the appropriate JAR's native library (SO or DLL) does not load and results into link error, then 'load' as above.

https://netty.io/wiki/forked-tomcat-native.html