iOS 14.5中的CoreML内存泄漏 [英] CoreML Memory Leak in iOS 14.5

问题描述

在我的应用程序中，我将VNImageRequestHandler与自定义MLModel一起用于对象检测.

In my application, I used VNImageRequestHandler with a custom MLModel for object detection.

该应用程序可以在14.5之前的iOS版本上正常运行.

The app works fine with iOS versions before 14.5.

iOS 14.5出现时，它破坏了一切.

When iOS 14.5 came, it broke everything.

1. 每当 try handler.perform([visionRequest])引发错误时(Error Domain = com.apple.vis Code = 11遇到未知异常" UserInfo = {NSLocalizedDescription =遇到未知异常})， pixelBuffer 内存将被保存并且永远不会释放，它使AVCaptureOutput的缓冲区已满，然后没有新帧出现.
2. 我必须按如下所示更改代码，通过将pixelBuffer复制到另一个var中，我解决了新帧没有到来但仍然发生内存泄漏问题的问题.

1. Whenever try handler.perform([visionRequest]) throws an error (Error Domain=com.apple.vis Code=11 "encountered unknown exception" UserInfo={NSLocalizedDescription=encountered unknown exception}), the pixelBuffer memory is held and never released, it made the buffers of AVCaptureOutput full then new frame not came.
2. I have to change the code as below, by copy the pixelBuffer to another var, I solved the problem that new frame not coming, but memory leak problem is still happened.

由于内存泄漏，该应用在一段时间后崩溃了.

Because of memory leak, the app crashed after some times.

请注意，在iOS版本14.5之前，检测可以完美运行， try handler.perform([visionRequest])绝不会引发任何错误.

Notice that before iOS version 14.5, detection works perfectly, try handler.perform([visionRequest]) never throws any error.

这是我的代码:

private func predictWithPixelBuffer(sampleBuffer: CMSampleBuffer) {
  guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
  }
  
  // Get additional info from the camera.
  var options: [VNImageOption : Any] = [:]
  if let cameraIntrinsicMatrix = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {
    options[.cameraIntrinsics] = cameraIntrinsicMatrix
  }
  
  autoreleasepool {
    // Because of iOS 14.5, there is a bug that when perform vision request failed, pixel buffer memory leaked so the AVCaptureOutput buffers is full, it will not output new frame any more, this is a temporary work around to copy pixel buffer to a new buffer, this currently make the memory increased a lot also. Need to find a better way
    var clonePixelBuffer: CVPixelBuffer? = pixelBuffer.copy()
    let handler = VNImageRequestHandler(cvPixelBuffer: clonePixelBuffer!, orientation: orientation, options: options)
    print("[DEBUG] detecting...")
    
    do {
      try handler.perform([visionRequest])
    } catch {
      delegate?.detector(didOutputBoundingBox: [])
      failedCount += 1
      print("[DEBUG] detect failed \(failedCount)")
      print("Failed to perform Vision request: \(error)")
    }
    clonePixelBuffer = nil
  }
}

有人遇到过同样的问题吗?如果是这样，您如何解决它?

Has anyone experienced the same problem? If so, how did you fix it?

推荐答案

我使用@Matthijs Hollemans CoreMLHelpers库对此进行了部分修复.

I have a partial fix for this using @Matthijs Hollemans CoreMLHelpers library.

我使用的模型有300个类别和2363个锚点.我使用了Matthijs在此处提供的许多代码将模型转换为MLModel.

The model I use has 300 classes and 2363 anchors. I used a lot of the code Matthijs provided here to convert the model to MLModel.

最后一步，使用3个子模型构建管道:raw_ssd_output，解码器和nms.对于这种解决方法，您需要从管道中删除 nms 模型，并输出 raw_confidence 和 raw_coordinates .

In the last step a pipeline is built using the 3 sub models: raw_ssd_output, decoder, and nms. For this workaround you need to remove the nms model from the pipeline, and output raw_confidence and raw_coordinates.

在您的应用中，您需要从 CoreMLHelpers 添加代码.

In your app you need to add the code from CoreMLHelpers.

然后添加此函数以解码MLModel的输出:

Then add this function to decode the output from your MLModel:

    func decodeResults(results:[VNCoreMLFeatureValueObservation]) -> [BoundingBox] {
        let raw_confidence: MLMultiArray = results[0].featureValue.multiArrayValue!
        let raw_coordinates: MLMultiArray = results[1].featureValue.multiArrayValue!
        print(raw_confidence.shape, raw_coordinates.shape)
        var boxes = [BoundingBox]()
        let startDecoding = Date()
        for anchor in 0..<raw_confidence.shape[0].int32Value {
            var maxInd:Int = 0
            var maxConf:Float = 0
            for score in 0..<raw_confidence.shape[1].int32Value {
                let key = [anchor, score] as [NSNumber]
                let prob = raw_confidence[key].floatValue
                if prob > maxConf {
                    maxInd = Int(score)
                    maxConf = prob
                }
            }
            let y0 = raw_coordinates[[anchor, 0] as [NSNumber]].doubleValue
            let x0 = raw_coordinates[[anchor, 1] as [NSNumber]].doubleValue
            let y1 = raw_coordinates[[anchor, 2] as [NSNumber]].doubleValue
            let x1 = raw_coordinates[[anchor, 3] as [NSNumber]].doubleValue
            let width = x1-x0
            let height = y1-y0
            let x = x0 + width/2
            let y = y0 + height/2
            let rect = CGRect(x: x, y: y, width: width, height: height)
            let box = BoundingBox(classIndex: maxInd, score: maxConf, rect: rect)
            boxes.append(box)
        }
        let finishDecoding = Date()
        let keepIndices = nonMaxSuppressionMultiClass(numClasses: raw_confidence.shape[1].intValue, boundingBoxes: boxes, scoreThreshold: 0.5, iouThreshold: 0.6, maxPerClass: 5, maxTotal: 10)
        let finishNMS = Date()
        var keepBoxes = [BoundingBox]()
        
        for index in keepIndices {
            keepBoxes.append(boxes[index])
        }
        print("Time Decoding", finishDecoding.timeIntervalSince(startDecoding))
        print("Time Performing NMS", finishNMS.timeIntervalSince(finishDecoding))
        return keepBoxes
    }

然后，当您从Vision接收结果时，您将调用以下函数:

Then when you receive the results from Vision, you call the function like this:

if let rawResults = vnRequest.results as? [VNCoreMLFeatureValueObservation] {
   let boxes = self.decodeResults(results: rawResults)
   print(boxes)
}

该解决方案很慢，原因是我移动数据并制定 BoundingBox 类型列表的方式.使用基础指针处理MLMultiArray数据将更加有效，并且可能使用Accelerate来找到每个锚点框的最大分数和最佳类别.

This solution is slow because of the way I move the data around and formulate my list of BoundingBox types. It would be much more efficient to process the MLMultiArray data using underlying pointers, and maybe use Accelerate to find the maximum score and best class for each anchor box.

这篇关于iOS 14.5中的CoreML内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！