How can I fix drawing overlapping 2D objects in Metal (stencil, clipping)?

I am currently making a simple 2D rendering application using Apple Metal 3 in C++. I want to draw overlapping 2D objects using stencil test, but it does not work.

A constructor for the Renderer class. Creating a stencil texture, descriptor and stencilState object.

Renderer::Renderer( MTL::Device* pDevice, MTK::View* pView )
{
  ...
  auto* pTexDesc = MTL::TextureDescriptor::alloc()->init();
  pTexDesc->setTextureType( MTL::TextureType2D );
  pTexDesc->setWidth( _viewWidth );
  pTexDesc->setHeight( _viewHeight );
  pTexDesc->setPixelFormat( MTL::PixelFormatStencil8 );
  pTexDesc->setStorageMode( MTL::StorageModePrivate );
  pTexDesc->setUsage( MTL::TextureUsageRenderTarget );
  _pStencilTexture = _pDevice->newTexture( pTexDesc );

  _pDepthStencilDesc = MTL::DepthStencilDescriptor::alloc()->init();
  _pStencilState = _pDevice->newDepthStencilState( _pDepthStencilDesc );
  ...
}

Update method(called every frame).

void Renderer::update( float delta )
{
  NS::AutoreleasePool* pPool = NS::AutoreleasePool::alloc()->init();
  _pCurrentDrawableView->setClearColor( MTL::ClearColor::Make(1.0f, 1.0f, 1.0f, 1.0f) );

  MTL::CommandBuffer* pCmd = _pCommandQueue->commandBuffer();
  MTL::RenderPassDescriptor* pRPD = _pCurrentDrawableView->currentRenderPassDescriptor();

  auto* pStencilAttach = pRPD->stencilAttachment();
  pStencilAttach->setTexture( _pStencilTexture );
  pStencilAttach->setClearStencil( 0 );
  pStencilAttach->setLoadAction( MTL::LoadActionClear );
  pStencilAttach->setStoreAction( MTL::StoreActionStore );

  MTL::RenderCommandEncoder* pEnc = pCmd->renderCommandEncoder(pRPD);
  
  // ADD WHAT YOU WANT TO DRAW
  pEnc->setDepthStencilState( _pStencilState );

  auto* pStencil = MTL::StencilDescriptor::alloc()->init();
  _pDepthStencilDesc->setFrontFaceStencil( pStencil );
  _pDepthStencilDesc->setBackFaceStencil( pStencil );

  { // draw orc
    pStencil->setStencilCompareFunction( MTL::CompareFunctionAlways ); // glStencilFunc(func, _, _);
    pEnc->setStencilReferenceValue( 1 ); // glStencilFunc(_, ref, _);
    pStencil->setWriteMask( 0xff ); // glStencilFunc(_, _, mask);
    pStencil->setStencilFailureOperation( MTL::StencilOperationReplace ); // glStencilOp( _, _, zpass );
    
    orc->update( delta );
    drawImage( orc->texture(), { 0.0f, 0.0f }, { 100.0f, 100.0f }, orc->frame() );
    drawFrame( pCmd, pEnc );
  }

  { // draw grass backgroud
    pStencil->setStencilCompareFunction( MTL::CompareFunctionEqual ); // glStencilFunc(func, _, _);
    pEnc->setStencilReferenceValue( 0 ); // glStencilFunc(_, ref, _);
    pStencil->setWriteMask( 0x00 ); // glStencilFunc(_, _, mask); 
    pStencil->setStencilFailureOperation( MTL::StencilOperationKeep ); // glStencilOp( _, _, zpass );

    _pCurrentTexture = _textureMap["Grass"]->_pTexture;
    fillRect( { 0.0f, 0.0f, _viewWidth / 2.0f, _viewHeight / 2.0f } );
    drawFrame( pCmd, pEnc );
  }

  pEnc->endEncoding();
  pCmd->presentDrawable(_pCurrentDrawableView->currentDrawable());
  pCmd->commit();

  pStencil->release();
  pPool->release();
}

*drawImage and fillRect is a kind of method that pushes vertices and indices information.

drawFrame

void Renderer::drawFrame( MTL::CommandBuffer* pCmd, MTL::RenderCommandEncoder* pEnc )
{
  NS::AutoreleasePool* pPool = NS::AutoreleasePool::alloc()->init();

  pEnc->setRenderPipelineState(_pPSO);

  _pVertexBuffer = _pDevice->newBuffer(_vertices.data(), sizeof(shader_types::VertexData) * _vertices.size(), MTL::StorageModeManaged);
  _pIndexBuffer = _pDevice->newBuffer(_indices.data(), sizeof(uint) * _indices.size(), MTL::StorageModeManaged);

  pEnc->setVertexBuffer(_pVertexBuffer, /* offset */ 0, /* index */ 0);
  pEnc->setFragmentTexture(_pCurrentTexture, 0);

  pEnc->drawIndexedPrimitives(MTL::PrimitiveTypeTriangle,
                            _pIndexBuffer->length(), MTL::IndexTypeUInt16,
                            _pIndexBuffer, 0);

  // flush vectors
  _vertices.clear();
  _indices.clear();
  _indexBufferIndex = 0;

  pPool->release();
}

I am not using the depth test to simplify the code. The result of this code is drawing only background. (Metal draws the last elements) As far as I know, the content of the orc pixels should be 1 and the content of the background pixels should be 0. Please give me any advice. Anything would be helpful to me. Thanks a lot.

I have tried to change an order to create and set variables in update(delta) method. But it does not work either. Also, I am more familiar with OpenGL than Metal. It seems like Metal API are not matched in OpenGL's API.

You are misunderstanding how states and descriptors work.

MTLDepthStencilState is an immutable object, not like OpenGL states, which means that you can't change it after the fact, even if you change the descriptor. So you need to create a proper descriptor before you call newDepthStencilState.

Also, Property frontFaceStencil and backFaceStencil are defined as @property (copy, nonatomic, null_resettable) in Metal headers:

@property (copy, nonatomic, null_resettable) MTLStencilDescriptor *frontFaceStencil;
@property (copy, nonatomic, null_resettable) MTLStencilDescriptor *backFaceStencil;

This means the MTLStencilDescriptor inside MTLDepthStencilDescriptor is copied by value, and not set by reference.

All of this means that in order for update to work, you need to write Renderer::Renderer like this:

_pOrcDepthStencilDesc = MTL::DepthStencilDescriptor::alloc()->init();
// Set up orc depth stencil state
auto* pOrcStencil = MTL::StencilDescriptor::alloc()->init();
pOrcStencil->setStencilCompareFunction( MTL::CompareFunctionAlways ); // glStencilFunc(func, _, _);
pOrcStencil->setWriteMask( 0xff ); // glStencilFunc(_, _, mask);
pOrcStencil->setStencilFailureOperation( MTL::StencilOperationReplace ); // glStencilOp( _, _, zpass );
_pOrcDepthStencilDesc->setFrontFacingState(...);
_pOrcDepthStencilDesc->setBackFacingState(...);
_pOrcStencilState = _pDevice->newDepthStencilState( _pOrcDepthStencilDesc );
_pGrassDepthStencilDesc = MTL::DepthStencilDescriptor::alloc()->init();
// Set up grass depth stencil state
_pGrassDepthStencilDesc->setFrontFacingState(...);
_pGrassDepthStencilDesc->setBackFacingState(...);
_pGrassStencilState = _pDevice->newDepthStencilState( _pGrassDepthStencilDesc );

and then in update:

{ // draw orc
pEnc->setDepthStencilState( _pOrcStencilState );
pEnc->setStencilReferenceValue( 1 ); // glStencilFunc(_, ref, _);
orc->update( delta );
drawImage( orc->texture(), { 0.0f, 0.0f }, { 100.0f, 100.0f }, orc->frame() );
drawFrame( pCmd, pEnc );
}

I found a solution. I just misunderstood how the stencil test works.

Building a texture for stencil test at the initializing point of the Renderer.

// build a texture for stencil test
void Renderer::buildStencilTest()
{
  auto* pStencilDesc = MTL::TextureDescriptor::texture2DDescriptor(
      MTL::PixelFormatDepth32Float_Stencil8, _viewSize[0], _viewSize[1], false);
  pStencilDesc->setStorageMode( MTL::StorageModePrivate );
  pStencilDesc->setUsage( MTL::TextureUsageRenderTarget | MTL::TextureUsageShaderRead );
  _pDepthStencilTexture = _pDevice->newTexture( pStencilDesc );
}

In the update() method, set the texture what we built for stencil test to the stencil attachment of current render pass of a view.

1. Clear stencil value as ZERO.

2. Draw the area what I want to mask. (I am not gonna use Depth test, so disable it.)

3. Set the stencil compare function to "Pass Always" and depth/stencil pass operation as "Replace".

4. Set stencil reference value to 1. Stencil test is gonna compare this and the value of stencil texture. At this time, stencil test "always PASS" and replace the drawing area of texture as the result of logical AND of "reference value (1)" and "write mask (0xff)" therefore 1.

5. Draw elements.

void Renderer::update() {
  // Clear stencil 
  pRPD->stencilAttachment()->setLoadAction( MTL::LoadActionClear );
  pRPD->stencilAttachment()->setStoreAction( MTL::StoreActionStore );
  pRPD->stencilAttachment()->setTexture( _pDepthStencilTexture );
  pRPD->stencilAttachment()->setClearStencil( 0 );

  auto* pDsDesc = MTL::DepthStencilDescriptor::alloc()->init();
  pDsDesc->setDepthCompareFunction( MTL::CompareFunctionAlways );
  pDsDesc->setDepthWriteEnabled( false );
  auto* pStencil = MTL::StencilDescriptor::alloc()->init();
  
  pStencil->setStencilCompareFunction(MTL::CompareFunctionAlways);

  pStencil->setDepthFailureOperation( MTL::StencilOperationKeep );
  pStencil->setStencilFailureOperation( MTL::StencilOperationKeep );
  pStencil->setDepthStencilPassOperation( MTL::StencilOperationReplace );
  pStencil->setReadMask(0xff);
  pStencil->setWriteMask(0xff);

  pDsDesc->setFrontFaceStencil(pStencil);
  pDsDesc->setBackFaceStencil(pStencil);
  pEnc->setStencilReferenceValue(1);
  pEnc->setDepthStencilState( _pDevice->newDepthStencilState( pDsDesc ) );

  drawImage( _orcTexture, { 0.0f, 0.0f }, { 50.0f, 50.0f }, _currentFrame );
  drawFrame(pEnc);
  ... 
}

Drawing the other area which is gonna be the background.

1. Set stencil compare function to "Equal".

2. Set depth/stencil pass operation to "Keep". If the stencil test failed, the area of stencil texture is gonna be 0.

3. Draw elements.

void Renderer::update() {
  pDsDesc->setDepthCompareFunction( MTL::CompareFunctionAlways );
  pDsDesc->setDepthWriteEnabled( false );

  pStencil->setStencilCompareFunction(MTL::CompareFunctionEqual);

  pStencil->setDepthFailureOperation(MTL::StencilOperationKeep);
  pStencil->setStencilFailureOperation(MTL::StencilOperationKeep);
  pStencil->setDepthStencilPassOperation(MTL::StencilOperationKeep);
  pStencil->setReadMask(0xff);
  pStencil->setWriteMask(0xff);

  pDsDesc->setFrontFaceStencil(pStencil);
  pDsDesc->setBackFaceStencil(pStencil);
  pEnc->setDepthStencilState( _pDevice->newDepthStencilState( pDsDesc ) );

  drawImage( grassTexture, { 0.0f, 0.0f }, { 200.0f, 200.0f } );
  drawFrame( pEnc );
  ... 
}

It will draw only the small masked area with the masked texture(grass) of second elements.