使用AVCaptureSession& amp;翻转相机时进行无缝音频录制AVAssetWriter [英] Seamless audio recording while flipping camera, using AVCaptureSession & AVAssetWriter

问题描述

我正在寻找一种在前后摄像头之间切换时保持无缝音轨的方法.市场上许多应用程序都可以做到这一点，例如SnapChat ...

I’m looking for a way to maintain a seamless audio track while flipping between front and back camera. Many apps in the market can do this, one example is SnapChat…

解决方案应使用AVCaptureSession和AVAssetWriter.此外，由于存在 bug .另外，我负担不起后期处理时间.

Solutions should use AVCaptureSession and AVAssetWriter. Also it should explicitly not use AVMutableComposition since there is a bug between AVMutableComposition and AVCaptureSession ATM. Also, I can't afford post processing time.

当前，当我更改视频输入时，音频记录会跳过并变得不同步.

Currently when I change the video input the audio recording skips and becomes out of sync.

我包括了可能相关的代码.

I’m including the code that could be relevant.

翻转相机

-(void) updateCameraDirection:(CamDirection)vCameraDirection {
    if(session) {
        AVCaptureDeviceInput* currentInput;
        AVCaptureDeviceInput* newInput;
        BOOL videoMirrored = NO;
        switch (vCameraDirection) {
            case CamDirection_Front:
                currentInput = input_Back;
                newInput = input_Front;
                videoMirrored = NO;
                break;
            case CamDirection_Back:
                currentInput = input_Front;
                newInput = input_Back;
                videoMirrored = YES;
                break;
            default:
                break;
        }

        [session beginConfiguration];
        //disconnect old input
        [session removeInput:currentInput];
        //connect new input
        [session addInput:newInput];
        //get new data connection and config
        dataOutputVideoConnection = [dataOutputVideo connectionWithMediaType:AVMediaTypeVideo];
        dataOutputVideoConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
        dataOutputVideoConnection.videoMirrored = videoMirrored;
        //finish
        [session commitConfiguration];
    }
}

样本缓冲区

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    //not active
    if(!recordingVideo)
        return;

    //start session if not started
    if(!startedSession) {
        startedSession = YES;
        [assetWriter startSessionAtSourceTime:CMSampleBufferGetPresentationTimeStamp(sampleBuffer)];
    }

    //Process sample buffers
    if (connection == dataOutputAudioConnection) {
        if([assetWriterInputAudio isReadyForMoreMediaData]) {
            BOOL success = [assetWriterInputAudio appendSampleBuffer:sampleBuffer];
            //…
        }

    } else if (connection == dataOutputVideoConnection) {
        if([assetWriterInputVideo isReadyForMoreMediaData]) {        
            BOOL success = [assetWriterInputVideo appendSampleBuffer:sampleBuffer];
            //…
        }
    }
}

也许调整音频采样的时间戳吗?

Perhaps adjust audio sample timeStamp?

推荐答案

嘿，我遇到了同样的问题，并发现在切换相机后，下一帧被推离了适当的位置.此后似乎会移动每一帧，从而导致视频和音频不同步.我的解决方案是在切换相机后将所有错位的帧移动到正确的位置.

Hey I was facing the same issue and discovered that after switching cameras the next frame was pushed far out of place. This seemed to shift every frame after that thus causing the the video and audio to be out of sync. My solution was to shift every misplaced frame to it's correct position after switching cameras.

很抱歉，我的答案将是Swift 4.2

您必须使用AVAssetWriterInputPixelBufferAdaptor才能在指定的演示时间戳记上附加示例缓冲区.

You'll have to use AVAssetWriterInputPixelBufferAdaptor in order to append the sample buffers at a specify presentation timestamp.

previousPresentationTimeStamp是上一帧的显示时间戳，而currentPresentationTimestamp是您猜测当前帧的显示时间戳. maxFrameDistance在测试时效果很好，但是您可以根据自己的喜好进行更改.

previousPresentationTimeStamp is the presentation timestamp of the previous frame and currentPresentationTimestamp is as you guessed the presentation timestamp of the current. maxFrameDistance worked every well when testing but you can change this to your liking.

let currentFramePosition = (Double(self.frameRate) * Double(currentPresentationTimestamp.value)) / Double(currentPresentationTimestamp.timescale)
let previousFramePosition = (Double(self.frameRate) * Double(previousPresentationTimeStamp.value)) / Double(previousPresentationTimeStamp.timescale)
var presentationTimeStamp = currentPresentationTimestamp
let maxFrameDistance = 1.1
let frameDistance = currentFramePosition - previousFramePosition
if frameDistance > maxFrameDistance {
    let expectedFramePosition = previousFramePosition + 1.0
    //print("[mwCamera]: Frame at incorrect position moving from \(currentFramePosition) to \(expectedFramePosition)")

    let newFramePosition = ((expectedFramePosition) * Double(currentPresentationTimestamp.timescale)) / Double(self.frameRate)

    let newPresentationTimeStamp = CMTime.init(value: CMTimeValue(newFramePosition), timescale: currentPresentationTimestamp.timescale)

    presentationTimeStamp = newPresentationTimeStamp
}

let success = assetWriterInputPixelBufferAdator.append(pixelBuffer, withPresentationTime: presentationTimeStamp)
if !success, let error = assetWriter.error {
    fatalError(error.localizedDescription)
}

也请注意-之所以有效，是因为我保持帧速率一致，因此请确保在此过程中您完全控制了捕获设备的帧速率.

Also please note - This worked because I kept the frame rate consistent, so make sure that you have total control of the capture device's frame rate throughout this process.

我在此处使用此逻辑进行了存储库

这篇关于使用AVCaptureSession& amp;翻转相机时进行无缝音频录制AVAssetWriter的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！