The image buffer stops getting produced after a few times

I'm using the Core Image framework to detect smiles in the VIPER architecture.

ViewController

extension ViewController: AVCaptureDataOutputSynchronizerDelegate {
    func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
        guard let syncedDepthData: AVCaptureSynchronizedDepthData =
                synchronizedDataCollection.synchronizedData(for: multifaceCamera.depthDataOutput) as? AVCaptureSynchronizedDepthData,
              let videoDataOutput = self.videoDataOutput,
              let syncedVideoData: AVCaptureSynchronizedSampleBufferData =
                synchronizedDataCollection.synchronizedData(for: videoDataOutput) as? AVCaptureSynchronizedSampleBufferData else {
            return
        }
        
        if syncedDepthData.depthDataWasDropped || syncedVideoData.sampleBufferWasDropped {
            return
        }
        
        let depthData = syncedDepthData.depthData
        let _depthData = depthData.converting(toDepthDataType: kCVPixelFormatType_DepthFloat16)
        let depthPixelBuffer = _depthData.depthDataMap
        let sampleBuffer = syncedVideoData.sampleBuffer
        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
            return
        }

        Task {
            await presenter.sendOutput(output: imageBuffer, depthPixelBuffer: depthPixelBuffer)
        }
    }
}

The view controller captures the sample buffer and the depth data and sends them over to the presenter. The depth data is from the TrueDepth camera, but the use of it is not shown in this example code for irrelevancy.

Presenter

func sendOutput(output: CVPixelBuffer, depthPixelBuffer: CVPixelBuffer?) async {
    await interactor.sendOutput(output: output, depthPixelBuffer: depthPixelBuffer)
}

The sendOutput method in the presenter class relays the data over to the interactor.

Interactor

func sendOutput(output: CVPixelBuffer, depthPixelBuffer: CVPixelBuffer?) async {
    let image = CIImage(cvPixelBuffer: output)
    let result = await isSmiling(image: image)
    /// Do something the result
}

func isSmiling(image: CIImage) async -> Bool {
    let detector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: [CIDetectorAccuracy: CIDetectorAccuracyHigh, CIDetectorSmile: true])
    let faces = detector?.features(in: image, options: [CIDetectorSmile : true as AnyObject]) as? [CIFaceFeature]
    if let facesFound = faces {
        for face in facesFound {
            return face.hasSmile
        }
    } else {
        return false
    }
    
    return false
}

The CIDetector in the interactor class uses the image buffer sent from the presenter to detect the smile status in faces. The problem I'm experiencing is that detector?.features(in: image, options: [CIDetectorSmile : true as AnyObject]) as? [CIFaceFeature] doesn't produce any results. I don't mean that it produces nils. The method is called a few times then it simply stops as if realizing that the feature method is not producing any results or the task is being completed. Doesn't crash, doesn't gracefully produce nils or errors, the image buffer simply stops getting produced from the view controller. Logging shows that the detector object is created properly let detector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: [CIDetectorAccuracy: CIDetectorAccuracyHigh, CIDetectorSmile: true]).

Just to test that the image buffer is being created properly, I tried using Vision's VNDetectFaceLandmarksRequest to detect faces and it turns out this works perfectly.

What's confusing is that if I use the same isSmiling method in the view controller, it works fine as shown in the following example, which makes me wonder if this is related to Task and async-await (although Vision's face detection worked fine in the interactor).

ViewController

extension ViewController: AVCaptureDataOutputSynchronizerDelegate {
    func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection) {
        guard let syncedDepthData: AVCaptureSynchronizedDepthData =
                synchronizedDataCollection.synchronizedData(for: multifaceCamera.depthDataOutput) as? AVCaptureSynchronizedDepthData,
              let videoDataOutput = self.videoDataOutput,
              let syncedVideoData: AVCaptureSynchronizedSampleBufferData =
                synchronizedDataCollection.synchronizedData(for: videoDataOutput) as? AVCaptureSynchronizedSampleBufferData else {
            return
        }
        
        if syncedDepthData.depthDataWasDropped || syncedVideoData.sampleBufferWasDropped {
            return
        }
        
        let depthData = syncedDepthData.depthData
        let _depthData = depthData.converting(toDepthDataType: kCVPixelFormatType_DepthFloat16)
        let depthPixelBuffer = _depthData.depthDataMap
        let sampleBuffer = syncedVideoData.sampleBuffer
        guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
            return
        }

        Task {
           let image = CIImage(cvPixelBuffer: imageBuffer)
           let result = await isSmiling(image: image)
           /// Do something the result
        }
    }
}

The only error message I'm getting in the Xcode console is:
>The input tensor width must be aligned to 64-byte.

which I'm not even sure if it's related to CIDetector.

I tried setting AVCaptureVideoDataOutput as following

videoDataOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_64RGBALE)]
/// or kCVPixelFormatType_64RGBALE

or even selecting an available format whose width is divisible by 16 or 64 and assigning it to the device's active format, thinking that, given that each pixel is 4 bytes, the width of the image in pixels needs to be a multiple of 16 for the total width in bytes to be a multiple of 64:

for format in availableFormats {
    let dimensions = CMVideoFormatDescriptionGetDimensions(format.formatDescription)
    if dimensions.width % 16 == 0 {
        byteAlignedFormats.append(format)
    }
}

/// assign one of the byteAlignedFormats to be the device's active format

let deviceInput = try AVCaptureDeviceInput(device: device)
try device.lockForConfiguration()
device.activeFormat = byteAlignedFormat
device.unlockForConfiguration()

Unfortunately, none of the above attempts have worked.

You are most likely running out of buffers.

AVFoundation only reserves a limited number of frame buffers for camera capture. When this pool is empty, it simply stops delivering new frames – without any warning or error... 🙄 And it will not restart automatically, even if buffers are available again.

The problem here is that you are doing the face detection async, which is a good idea in theory. However, since the detection takes longer than capturing a new frame, your async queue will fill up with detection tasks, each of them retaining the corresponding capture buffer. And as soon as the capture buffer pool is empty, the session stops capturing.

What you could do is, as soon as a new buffer arrives from the camera, check if the detector is still busy checking the previous frame for faces. If this is the case, you can discard the frame and check again when the next frame arrives. This way, buffers are freed immediately instead of being retained by the async queue.