How to debug a DLL that is being used from Java through JNI

Hello good people of StackOverflow. I know this question is going to be somewhat unclear, but I am really lost here and have no one with whom I can discuss this, I can only describe the scenario.

I am working for a company where they use C++ to create a library, a DLL file, this DLL is access through Java using JNI. I want to debug the DLL for memory leak issues for which I used Visual Studio Memory Profiler and Deleaker. But the problem is that it is just a DLL (not an executable) which is used from a Java process (worker process). When I do try debugging the Java worker process which is using the DLL the debugger keeps breaking in Java.exe and JVM.DLL process, and does not let me debug the application, so no tool is able to gather enough information about memory allocation and deallocation to suggest any memory leak. When I attach the Java process with Visual Studio, it debugger keeps breaking every second, as shown in the below screenshot.

I found a similar question here https://stackoverflow.com/questions/359851/debug-a-dll-from-java-java-jni-dll-app-dll-app-dll but it did not help.

How should one debug the memory leak in such scenarios? Are there any tools that can be used here?

> the debugger keeps breaking in Java.exe and JVM.DLL process

> debugger keeps breaking every second

From the stack trace in your screen shot, I'd say the break is happening in JIT-compiled code, which Visual Studio can not unwind through, so you get this broken stack trace.

The JVM uses implicit exception dispatch, whereby often a null check is omitted, and a memory access is just done. If an EXCEPTION_ACCESS_VIOLATION occurs as a result, it is translated into a Java Exception in the JVMs exception handler. (See also: https://shipilev.net/jvm/anatomy-quarks/25-implicit-null-checks/ for an extended description). The same is done in some other cases such as stack overflow, and divide by zero.

But, the debugger will also catch all these exceptions/signals first. I personally always just continue (g) through them until I get to the native code I want to debug. But for profiling purposes, this might be too cumbersome.

I think the most straight forward solution is to just disable the first chance exception handling for access violations in the debugger. I believe the right command would be sxi av. And do the same for other exceptions that keep showing up.

You should also be able to prevent the JVM's use of implicit null checks by using the -XX:+UnlockDiagnosticVMOptions -XX:-ImplicitNullChecks JVM flags. But IIRC this is not a 100% solution to prevent the debugger from stopping on JVM internal exceptions.

You can do it by adding a

__debugbreak();

in the Dll's code, just before the part you suspect to have a memory leak. You must compile your Dll in debug mode.

This __debugbreak is an intrinsic that will cause your java program (in fact the Dll) to be interrupted when it reaches this instruction, you will see a dialogbox asking if you want to debug the program, you must say yes, then you will be asked to select which debugger to use (Visual Studio).

So, you won't attach the process to the debugger, instead the debugger (Visual Studio) will be launched when the __debugbreak is reached.

If everything goes well, you will se the Dll's code and you will be able to step to the next instruction (F10) and debug as usual.