Debugging COM interface mapping in JNA

After mapping Vss.h and several others headers to Java/JNA (see this question) I am trying to run some of the COM object methods and have a problem debugging them.

I do not know if I am calling the correct method or one with similar parameters. Some of the error codes I find in the JNA documentation here, but it does not include all the errors I am seeing.

Some examples:

// gather writer metadata
public int GatherWriterMetadata(IVssAsync pAsync)
{
    return _invokeNativeInt( 5, new Object[] { getPointer(), pAsync });
}

I have error -2147212542

For

// Called to set the context for subsequent snapshot-related operations
public int SetContext(WinDef.LONG lContext)
{
    return _invokeNativeInt( 32, new Object[] { getPointer(), lContext });
}

I have
java.lang.Error: Invalid memory access
at com.sun.jna.Native.invokeInt(Native Method)

I've tried to play with a numbers like 31,32 and 33 for the SetContext method.

Do not try to "play with numbers" as you are likely to experience random behavior.

As I mentioned in my answer to your other question the integer vtblId value for the _invokeNative... calls has to come from the Vtbl structure in the header file. I don't have direct access to the header file, but this mapping from Rust is probably good to use, but since this interface (and all COM interfaces) extends IUnknown, it already includes the functions QueryInterface(), AddRef(), and Release(), which take up vtblId values 0, 1, and 2.

Your GatherWriterMetadata method, using a vtblId of 5, is actually invoking the InitializeForBackup() function, which expects a BSTR argument. You are giving it some other argument, so it is returning an error. (If looking up an error by the decimal value -2147212542 doesn't work, you can translate to two's-complement hex, which in this case is 0x80042302, a System Restore Error.)

By my count, you should be using vtblId of 9 for GatherWriterMetadata. Please count for yourself to confirm.

Your SetContext method, by my count should be using vtblId of 35. Again, please count the number of functions (starting at 3) to confirm this for yourself.

Also, I see you've used an int type for the return type for most of these functions rather than HRESULT. Since HRESULT is eventually a 32-bit integer type, this will work. However, if you actually use HRESULT as the return value you gain access to more convenient/self-documenting error-handling methods like COMUtils.SUCCEEDED() and COMUtils.FAILED(), or even the COMUtils.checkRC() method which throws a nicely formatted COMException on failure.

So your mappings should probably be:

// gather writer metadata
public HRESULT GatherWriterMetadata(IVssAsync pAsync)
{
    return _invokeNativeObject( 9,
        new Object[] { getPointer(), pAsync }, HRESULT.class);
}

and

// Called to set the context for subsequent snapshot-related operations
public HRESULT SetContext(WinDef.LONG lContext)
{
    return _invokeNativeObject( 35,
        new Object[] { getPointer(), lContext }, HRESULT.class);
}

Incidentally, since the Windows LONG type is always 32-bits, you could also simplify the second mapping to:

public HRESULT SetContext(int lContext) { ... }