结合CoreML和ARKit [英] Combining CoreML and ARKit

问题描述

我正在尝试使用Apple 网站.

我从ARKit(Xcode 9 beta 3)的标准模板开始

我不再使用新的摄影机会话，而是重用了ARSCNView启动的会话.

在viewDelegate的结尾，我写:

  sceneView.session.delegate =自我 

然后我将viewController扩展为符合ARSessionDelegate协议(可选协议)

 //MARK:ARSessionDelegate扩展ViewController:ARSessionDelegate {func session(_ session:ARSession，didUpdate frame:ARFrame){做 {让预测=尝试self.model.prediction(image:frame.capturedImage)DispatchQueue.main.async {如果让prob = projection.classLabelProbs [prediction.classLabel] {self.textLabel.text ="\(prediction.classLabel)\(String(描述:问题))"}}}捕获让错误作为NSError {print(发生意外错误:\(error.localizedDescription).")}}} 

起初，我尝试了该代码，但随后注意到初始需要Image类型的像素Buffer.<RGB< 299,299>.

尽管没有推荐，但我想我只是调整框架的大小，然后尝试从中进行预测.我正在使用此函数调整大小(取自 https://github.com/yulingtianxia/Core-ML-样本)

  func resize(pixelBuffer:CVPixelBuffer)->CVPixelBuffer?{让imageSide = 299var ciImage = CIImage(cvPixelBuffer:pixelBuffer，选项:无)let transform = CGAffineTransform(scaleX:CGFloat(imageSide)/CGFloat(CVPixelBufferGetWidth(pixelBuffer))，y:CGFloat(imageSide)/CGFloat(CVPixelBufferGetHeight(pixelBuffer)))ciImage = ciImage.transformed(by:transform).cropped(to:CGRect(x:0，y:0，width:imageSide，height:imageSide))让ciContext = CIContext()var resizeBuffer:CVPixelBuffer?CVPixelBufferCreate(kCFAllocatorDefault，imageSide，imageSide，CVPixelBufferGetPixelFormatType(pixelBuffer)，nil和resizeBuffer)ciContext.render(ciImage，to:resizeBuffer！)返回resizeBuffer} 

不幸的是，这还不足以使其正常工作.这是捕获的错误:

 发生意外错误:输入图像特征图像与模型描述不匹配.2017-07-20 AR + MLPhotoDuplicatePrediction [928:298214] [核心]错误Domain = com.apple.CoreML代码= 1输入图像特征图像与模型描述不匹配"UserInfo = {NSLocalizedDescription =输入图像特征图像与模型描述不匹配，NSUnderlyingError = 0x1c4a49fc0 {Error Domain = com.apple.CoreML代码= 1图像不应该是32-BGRA或32-ARGB类型，不支持(875704422)"UserInfo = {NSLocalizedDescription =不需要图像类型为32-BGRA或32-ARGB，而是不受支持(875704422)}}} 

不确定从这里可以做什么.

如果有更好的建议将两者结合起来，我会非常高兴.

编辑:我还尝试了苹果开发者论坛上运行.

Edit3 :所以我尝试实现rickster解决方案.与inceptionV3一起很好地工作.我想尝试一个功能观察(VNClassificationObservation).目前，它无法使用 TinyYolo .边界是错误的.试图弄清楚.

解决方案

不要自己处理图像以将其提供给Core ML.使用视觉.(不，不是那个.这一个.)视觉采用ML模型和几张图像中的任何一张类型(包括 CVPixelBuffer )并自动获取以正确的尺寸，宽高比和像素格式对图像进行评估，然后为您提供模型的结果.

这是您需要的代码的粗略骨架:

  var请求:VNRequestfunc setup(){让模型=尝试VNCoreMLModel(for:MyCoreMLGeneratedModelClass().model)request = VNCoreMLRequest(model:model，completeHandler:myResultsMethod)}func classifyARFrame(){让处理程序= VNImageRequestHandler(cvPixelBuffer:session.currentFrame.capturedImage，方向:.up)//根据您的UI方向进行修复handler.perform([request])}func myResultsMethod(请求:VNRequest，错误:错误?){守卫让结果= request.results为?[VNClassificationObservation]否则{fatalError("huh")}用于结果分类{print(classification.identifier，//场景标签分类(置信度)}} 

请参阅此答案以获取更多指针的另一个问题.

I am trying to combine CoreML and ARKit in my project using the given inceptionV3 model on Apple website.

I am starting from the standard template for ARKit (Xcode 9 beta 3)

Instead of intanciating a new camera session, I reuse the session that has been started by the ARSCNView.

At the end of my viewDelegate, I write:

sceneView.session.delegate = self

I then extend my viewController to conform to the ARSessionDelegate protocol (optional protocol)

// MARK: ARSessionDelegate
extension ViewController: ARSessionDelegate {

    func session(_ session: ARSession, didUpdate frame: ARFrame) {

        do {
            let prediction = try self.model.prediction(image: frame.capturedImage)
            DispatchQueue.main.async {
                if let prob = prediction.classLabelProbs[prediction.classLabel] {
                    self.textLabel.text = "\(prediction.classLabel) \(String(describing: prob))"
                }
            }
        }
        catch let error as NSError {
            print("Unexpected error ocurred: \(error.localizedDescription).")
        }
    }
}

At first I tried that code, but then noticed that inception requires a pixel Buffer of type Image. < RGB,<299,299>.

Although not recommenced, I thought I would just resize my frame then try to get a prediction out of it. I am resizing using this function (took it from https://github.com/yulingtianxia/Core-ML-Sample)

func resize(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    let imageSide = 299
    var ciImage = CIImage(cvPixelBuffer: pixelBuffer, options: nil)
    let transform = CGAffineTransform(scaleX: CGFloat(imageSide) / CGFloat(CVPixelBufferGetWidth(pixelBuffer)), y: CGFloat(imageSide) / CGFloat(CVPixelBufferGetHeight(pixelBuffer)))
    ciImage = ciImage.transformed(by: transform).cropped(to: CGRect(x: 0, y: 0, width: imageSide, height: imageSide))
    let ciContext = CIContext()
    var resizeBuffer: CVPixelBuffer?
    CVPixelBufferCreate(kCFAllocatorDefault, imageSide, imageSide, CVPixelBufferGetPixelFormatType(pixelBuffer), nil, &resizeBuffer)
    ciContext.render(ciImage, to: resizeBuffer!)
    return resizeBuffer
} 

Unfortunately, this is not enough to make it work. This is the error that is catched:

Unexpected error ocurred: Input image feature image does not match model description.
2017-07-20 AR+MLPhotoDuplicatePrediction[928:298214] [core] 
    Error Domain=com.apple.CoreML Code=1 
    "Input image feature image does not match model description" 
    UserInfo={NSLocalizedDescription=Input image feature image does not match model description, 
    NSUnderlyingError=0x1c4a49fc0 {Error Domain=com.apple.CoreML Code=1 
    "Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)" 
    UserInfo={NSLocalizedDescription=Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)}}}

Not sure what I can do from here.

If there is any better suggestion to combine both, I'm all ears.

Edit: I also tried the resizePixelBuffer method from the YOLO-CoreML-MPSNNGraph suggested by @dfd , the error is exactly the same.

Edit2: So I changed the pixel format to be kCVPixelFormatType_32BGRA (not the same format as the pixelBuffer passed in the resizePixelBuffer).

let pixelFormat = kCVPixelFormatType_32BGRA // line 48

I do not have the error anymore. But as soon as I try to make a prediction, the AVCaptureSession stops. Seems I am running into the same issue Enric_SA is running on the apple developers forum.

Edit3: So I tried implementing rickster solution. Works well with inceptionV3. I wanted to try a a feature observation (VNClassificationObservation). At this time, it is not working using TinyYolo. The bounding are wrong. Trying to figure it out.

解决方案

Don't process images yourself to feed them to Core ML. Use Vision. (No, not that one. This one.) Vision takes an ML model and any of several image types (including CVPixelBuffer) and automatically gets the image to the right size and aspect ratio and pixel format for the model to evaluate, then gives you the model's results.

Here's a rough skeleton of the code you'd need:

var request: VNRequest

func setup() {
    let model = try VNCoreMLModel(for: MyCoreMLGeneratedModelClass().model)
    request = VNCoreMLRequest(model: model, completionHandler: myResultsMethod)
}

func classifyARFrame() {
    let handler = VNImageRequestHandler(cvPixelBuffer: session.currentFrame.capturedImage,
        orientation: .up) // fix based on your UI orientation
    handler.perform([request])
}

func myResultsMethod(request: VNRequest, error: Error?) {
    guard let results = request.results as? [VNClassificationObservation]
        else { fatalError("huh") }
    for classification in results {
        print(classification.identifier, // the scene label
              classification.confidence)
    }
}

See this answer to another question for some more pointers.

这篇关于结合CoreML和ARKit的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！