与ARKIT一起拍摄视频 [英] Take a Video with ARKIT

问题描述

你好社区，

我尝试使用Swift 4和即将推出的 ARKit-Framework 构建应用程序，但是我陷入了困境.我需要使用框架或至少UIImage序列拍摄视频，但我不知道如何.

I try to build a App with Swift 4 and the great upcoming ARKit-Framework but I am stuck. I need to take a Video with the Framework or at least a UIImage-sequence but I dont know how.

这是我尝试过的:

在ARKit中，您有一个跟踪您的世界的会话.该会话具有一个 capturedImage 实例，您可以在其中获取当前图像.因此，我创建了一个计时器，该计时器将 capturedImage每0.1s 追加到一个列表中.这对我有用，但是如果我通过单击开始"按钮启动计时器，则相机开始滞后.我想这与计时器无关，因为如果我通过单击停止"按钮使计时器无效，则相机将再次流畅.

In ARKit you have a session which tracks your world. This session has a capturedImage instance where you can get the current Image. So I createt a Timer which appends the capturedImage every 0.1s to a List. This would work for me but If I start the Timer by clicking a "start"-button, the camera starts to lag. Its not about the Timer i guess because If I invalidate the Timer by clicking a "stop"-button the camera is fluent again.

有没有办法解决滞后甚至更好的办法?

Is there a way to solve the lags or even a better way?

谢谢

推荐答案

使用自定义渲染器.

使用自定义渲染器渲染场景，然后从自定义渲染器获取纹理，最后将其隐藏到CVPixelBufferRef

Render the scene using the custom renderer, then get texture from the custom renderer, finally covert that to a CVPixelBufferRef

- (void)viewDidLoad {
    [super viewDidLoad];

    self.rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    self.bytesPerPixel = 4;
    self.bitsPerComponent = 8;
    self.bitsPerPixel = 32;
    self.textureSizeX = 640;
    self.textureSizeY = 960;

    // Set the view's delegate
    self.sceneView.delegate = self;

    // Show statistics such as fps and timing information
    self.sceneView.showsStatistics = YES;

    // Create a new scene
    SCNScene *scene = [SCNScene scene];//[SCNScene sceneNamed:@"art.scnassets/ship.scn"];

    // Set the scene to the view
    self.sceneView.scene = scene;

    self.sceneView.preferredFramesPerSecond = 30;

    [self setupMetal];
    [self setupTexture];
    self.renderer.scene = self.sceneView.scene;

}

- (void)setupMetal
{
    if (self.sceneView.renderingAPI == SCNRenderingAPIMetal) {
        self.device = self.sceneView.device;
        self.commandQueue = [self.device newCommandQueue];
        self.renderer = [SCNRenderer rendererWithDevice:self.device options:nil];
    }
    else {
        NSAssert(nil, @"Only Support Metal");
    }
}

- (void)setupTexture
{
    MTLTextureDescriptor *descriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm_sRGB width:self.textureSizeX height:self.textureSizeY mipmapped:NO];
    descriptor.usage = MTLTextureUsageShaderRead | MTLTextureUsageRenderTarget;

    id<MTLTexture> textureA = [self.device newTextureWithDescriptor:descriptor];
    self.offscreenTexture = textureA;
}

- (void)renderer:(id <SCNSceneRenderer>)renderer willRenderScene:(SCNScene *)scene atTime:(NSTimeInterval)time
{
    [self doRender];
}

- (void)doRender
{
    if (self.rendering) {
        return;
    }
    self.rendering = YES;
    CGRect viewport = CGRectMake(0, 0, self.textureSizeX, self.textureSizeY);

    id<MTLTexture> texture = self.offscreenTexture;

    MTLRenderPassDescriptor *renderPassDescriptor = [MTLRenderPassDescriptor new];
    renderPassDescriptor.colorAttachments[0].texture = texture;
    renderPassDescriptor.colorAttachments[0].loadAction = MTLLoadActionClear;
    renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1.0);
    renderPassDescriptor.colorAttachments[0].storeAction = MTLStoreActionStore;

    id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];

    self.renderer.pointOfView = self.sceneView.pointOfView;

    [self.renderer renderAtTime:0 viewport:viewport commandBuffer:commandBuffer passDescriptor:renderPassDescriptor];

    [commandBuffer addCompletedHandler:^(id<MTLCommandBuffer> _Nonnull bf) {
        [self.recorder writeFrameForTexture:texture];
        self.rendering = NO;
    }];

    [commandBuffer commit];
}

然后在记录器中，用AVAssetWriter设置AVAssetWriterInputPixelBufferAdaptor.并将纹理转换为CVPixelBufferRef:

Then in the recorder, set up the AVAssetWriterInputPixelBufferAdaptor with AVAssetWriter. And convert the texture to CVPixelBufferRef:

- (void)writeFrameForTexture:(id<MTLTexture>)texture {
    CVPixelBufferPoolRef pixelBufferPool = self.assetWriterPixelBufferInput.pixelBufferPool;
    CVPixelBufferRef pixelBuffer;
    CVReturn status = CVPixelBufferPoolCreatePixelBuffer(nil, pixelBufferPool, &pixelBuffer);
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
    void *pixelBufferBytes = CVPixelBufferGetBaseAddress(pixelBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
    MTLRegion region = MTLRegionMake2D(0, 0, texture.width, texture.height);
    [texture getBytes:pixelBufferBytes bytesPerRow:bytesPerRow fromRegion:region mipmapLevel:0];

    [self.assetWriterPixelBufferInput appendPixelBuffer:pixelBuffer withPresentationTime:presentationTime];
    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
    CVPixelBufferRelease(pixelBuffer);
}

确保自定义渲染器和适配器共享相同的像素编码.

Make sure the custom renderer and the adaptor share the same pixel encoding.

我对此进行了默认的ship.scn测试，它只消耗30％的CPU，而相比之下，每帧使用snapshot方法则消耗了90％的CPU.而且这不会弹出一个权限对话框.

I tested this for the default ship.scn and it and it only consume 30% CPU compared to almost 90% compared to use snapshot method for every frame. And this will not pop up a permission dialog.

这篇关于与ARKIT一起拍摄视频的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！