iOS:处理来自AVPlayer视频轨道的音频 [英] iOS: Process audio from AVPlayer video track

问题描述

我计划在我的iOS应用中重构我的录音系统. 语境: 到目前为止，我分别录制视频和音频，大约同时开始录制.录制完成后，在同一系统上，我将分别播放视频和音频，并在音频上动态应用AudioUnits.最后，我合并了视频和修改后的音频. 碰巧两个记录都不同时开始(出于某种原因)，从而产生了不同步的结果.

I plan to refactor my recording system in My iOS app. Context: Up to now, I record video and audio separately, starting recording both approximatly at same time. Once record is finished, same system, I play the video and audio separately, applying AudioUnits on the fly on audio. Finally, I merge the video and modified audio. It happens that both records don't start at the same time (for any reasons), producing an unsynchronized result.

是否可以像这样重构我的系统:

Would it be possible to refactor my system like this:

1) Record normal video with audio into mov file --> I would be sure that audio+video would be synchronized.

2) During viewing the result with AVPlayer, process the audio part on the fly. (I will use AudioKit) --> that's the part I m not confident. 
   Would I be able to send the audio buffer to Audiokit (which would process it) and give back the processed audio to AVPlayer like if it was the original AVPlayer audio part?

3) Save a final file with video and audio modified --> easy part with AVFundation

请询问任何信息;)

推荐答案

我可以想到一种相当简单的方法.

I can think of one fairly simple way to do this.

基本上，您只需要在AKPlayer实例中打开视频文件即可.然后，您将视频音频静音.现在，您已经在AudioKit中获得了视频音频.使用公共时钟将视频和音频锁定在一起非常简单.流的伪代码:

Basically you just need to open your video file in an AKPlayer instance. Then, you mute your video audio. Now, you have the video audio in AudioKit. It's pretty simple to lock the video and audio together using a common clock. Pseudo-code of the flow:

// This will represent a common clock using the host time
var audioClock = CMClockGetHostTimeClock()

// your video player
let videoPlayer = AVPlayer( url: videoURL )
videoPlayer.masterClock = audioClock
videoPlayer.automaticallyWaitsToMinimizeStalling = false

....

var audioPlayer: AKPlayer?

// your video-audio player
if let player = try? AKPlayer(url: videoURL) {
    audioPlayer = player
}

func schedulePlayback(videoTime: TimeInterval, audioTime: TimeInterval, hostTime: UInt64 ) {
    audioPlay( audioTime, hostTime: hostTime )
    videoPlay(at: 0, hostTime: hostTime)
}


func audioPlay(at time: TimeInterval = 0, hostTime: UInt64 = 0) {
    audioPlayer.play(when: time, hostTime: hostTime)    
}

func videoPlay(at time: TimeInterval = 0, hostTime: UInt64 = 0 ) {
    let cmHostTime = CMClockMakeHostTimeFromSystemUnits(hostTime)
    let cmVTime = CMTimeMakeWithSeconds(time, 1000000)
    let futureTime = CMTimeAdd(cmHostTime, cmVTime)
    videoPlayer.setRate(1, time: kCMTimeInvalid, atHostTime: futureTime)
}

您可以通过正常方式将播放器连接到任何AudioKit处理链.

You can connect the player up to any AudioKit processing chain in the normal way.

要导出音频时，请在最终输出处理链上运行AKNodeRecorder.将其记录到文件中，然后将音频合并到视频中.我不确定正在处理的AudioKit离线处理是否准备就绪，因此您可能需要实时播放音频以捕获处理输出.

When you want to export your audio, run an AKNodeRecorder on the final output processing chain. Record this to file, then merge your audio into your video. I'm not sure if the AudioKit offline processing that is being worked on is ready yet, so you may need to play the audio in real time to capture the processing output.

这篇关于iOS:处理来自AVPlayer视频轨道的音频的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！