MediaFoundation: Registering custom ClassFactory doesn't work

Background:
I'm encoding NV12 buffers into h264 video stream wrapped in MPEG4 container using SinkWriter. Everything works fine but one problem though, as the SinkWriter abstracts the low-level encoder configurations I'm unable to control the properties like GOP size, B-picture count, CODECAPI_AVEncCommonRateControlMode, etc.

The problem is due to the fact that the SinkWriter instantiates the encoder transform only after the SetInputMediaType call, and we will be able to get the CodecAPI instance only after that point. So, we have no way to control the encoder and configure necessary props before all this happens, it also never honours further changes to the encoder through CodecAPI instance.

Experiments:
I tried the PropertyStore(MF_SINK_WRITER_ENCODER_CONFIG) approach but nothing seemed to be changing (It could be a platform/encoder specific behaviour), I could also see a lot of people complaining about unpredictable behaviours with these APIs. Then, I came across this MSDN thread (Almost 7 years old post) in which the user described how he dealt with this problem by locally registering a custom class factory on Windows7 machine.

Problem:
Having the MSDN thread as a reference, I tried implementing IClassFactory and registering it through MFTRegisterLocal but the CreateInstance function is never getting called for me (Windows 10 machine). I'm only getting the QueryInterface method called for IID_IClassFactory and IID_IMFAttributes interfaces instead. And, the SinkWriter seems to be proceeding with fetching the MFT on its own.

I understand I might be doing something wrong, and I'm not an expert in COM. Is there any other way I could achieve this?

<!-- language-all: c++ -->

Custom class factory implementation:

class MyClassFactory : public IClassFactory  {
public:
MyClassFactory () : _cRef(1) {}
~MyClassFactory() {}
// Only this method is getting called
STDMETHODIMP QueryInterface(REFIID riid, void** ppv) 
{
HRESULT hr = E_NOTIMPL;
// Only the below 2 cases (IID_IClassFactory and IID_IMFAttributes) are getting hit
if (IID_IClassFactory == riid) 
{
*ppv = static_cast<IClassFactory*>(this);
if (*ppv) {
reinterpret_cast<IUnknown*>(*ppv)->AddRef();
}
hr = S_OK;
}
else if (IID_IMFAttributes == riid) 
{
if (!pEncoder) {
hr = FindEncoderEx(&pEncoder);
}
IMFAttributes *attributes;
hr = pEncoder->GetAttributes(&attributes);
*ppv = attributes;
}
else 
{
//This case has never been reached
}
return hr;
}
//This is never called
STDMETHODIMP CreateInstance(IUnknown* pUnkOuter, REFIID riid, void** ppv)
{
HRESULT hr = S_OK;
if (pUnkOuter != NULL)
{
if (riid != __uuidof(IUnknown))
{
return E_NOINTERFACE;
}
}
if (!pEncoder) {
hr = FindEncoderEx(&pEncoder);
}
hr = pEncoder->QueryInterface(riid, ppv);
return hr;
}
IFACEMETHODIMP_(ULONG) AddRef()
{
return InterlockedIncrement(&_cRef);
}
IFACEMETHODIMP_(ULONG) Release()
{
assert(_cRef > 0);
LONG cRef = InterlockedDecrement(&_cRef);
if (!cRef)
delete this;
return cRef;
}
STDMETHODIMP LockServer(BOOL fLock) 
{
if (fLock) 
{
AddRef();
}
else {
Release();
}
return S_OK;
}
HRESULT FindEncoderEx(IMFTransform** ppEncoder)
{
...
}
protected:
LONG	_cRef;
CComPtr<IMFTransform> pEncoder = NULL;
};

Register custom class factory:

MyClassFactory* cf = new MyClassFactory();
MFT_REGISTER_TYPE_INFO infoInput = { MFMediaType_Video, MFVideoFormat_NV12 };
MFT_REGISTER_TYPE_INFO infoOutput = { MFMediaType_Video, MFVideoFormat_H264 };
MFTRegisterLocal(cf, MFT_CATEGORY_VIDEO_ENCODER, L"MyClassFactory", 0, 1, &infoInput, 1, &infoOutput);

Any help would be appreciated.

You are walking on thin ice here because what you are trying to do is not supposed to work. You do (or at least can) register a local transform but API would typically prefer other existing MFTs to your one because they are of higher internal merit (and with hardware assistance support), so you are not expected to override existing behavior.

Your real options are:

1. to use MF_SINK_WRITER_ENCODER_CONFIG attribute to pass encoder specific configuration
2. register class factory with COM rather than with MF for existing CLSID of encoder so that COM instantiation of MFT went your way; you would have to figure out details about COM to implement this and I would, in general, not recommend to interfere with standard behavior of COM registration/instantiation without good reason
3. run encoding MFT (or its equivalent - you don't have to use MFT exactly in this case) separately, outside of Sink Writer API, and feed Sink Writer with already compressed data