自定义AVVideoCompositing类无法按预期工作 [英] Custom AVVideoCompositing class not working as expected
问题描述
我正在尝试将CIFilter应用于AVAsset，然后在应用过滤器的情况下保存它。我这样做的方法是使用 AVAssetExportSession ，将 videoComposition 设置为 AVMutableVideoComposition 具有自定义 AVVideoCompositing 类的对象。
I'm attempting to apply a CIFilter to an AVAsset, and then save it with the filter applied. The way that I am doing this is by using an AVAssetExportSession with videoComposition set to an AVMutableVideoComposition object with a custom AVVideoCompositing class.
我也在设置 说明 我的 AVMutableVideoComposition 对象自定义组合指令类（符合 AVMutableVideoCompositionInstruction ）。这个类传递了一个跟踪ID，以及一些其他不重要的变量。
I am also setting the instructions of my AVMutableVideoComposition object to a custom composition instruction class (conforming to AVMutableVideoCompositionInstruction). This class is passed a track ID, along with a few other unimportant variables.
不幸的是，我遇到了一个问题 - startVideoCompositionRequest： 我的自定义视频合成器类中的功能（符合 AVVideoCompositing ）未正确调用。
Unfortunately, I've run into a problem - the startVideoCompositionRequest: function in my custom video compositor class (conforming to AVVideoCompositing) is not being called correctly.
当我将自定义指令类的 passthroughTrackID 变量设置为轨道ID时，startVideoCompositionRequest(request) AVVideoCompositing 中的c $ c> 功能未被调用。
When I set the passthroughTrackID variable of my custom instruction class to the track ID, the startVideoCompositionRequest(request) function in my AVVideoCompositing is not called.
然而，当我没有设置自定义指令类的 passthroughTrackID 变量时， startVideoCompositionRequest（request） 被调用，但不正确 - 打印 request.sourceTrackIDs 导致一个空数组， request.sourceFrameByTrackID（的TrackID） 导致零值。
Yet, when I do not set the passthroughTrackID variable of my custom instruction class, the startVideoCompositionRequest(request) is called, but not correctly - printing request.sourceTrackIDs results in an empty array, and request.sourceFrameByTrackID(trackID) results in a nil value.
我发现有趣的是 cancelAllPendingVideoCompositionRequests： 函数。它在 startVideoCompositionRequest：之前被调用一次，或者在 startVideoCompositionRequest：的情况下连续两次被调用没有被调用。
Something interesting that I found was that the cancelAllPendingVideoCompositionRequests: function is always called twice when attempting to export the video with filters. It is either called once before startVideoCompositionRequest: and once after, or just twice in a row in the case that startVideoCompositionRequest: is not called.
我创建了三个用于导出带过滤器的视频的类。这是实用程序类，它基本上只包含一个 export 函数并调用所有必需的代码
I've created three classes for exporting the video with filters. Here's the utility class, which basically just includes an export function and calls all of the required code
class VideoFilterExport{
let asset: AVAsset
init(asset: AVAsset){
self.asset = asset
}
func export(toURL url: NSURL, callback: (url: NSURL?) -> Void){
guard let track: AVAssetTrack = self.asset.tracksWithMediaType(AVMediaTypeVideo).first else{callback(url: nil); return}
let composition = AVMutableComposition()
let compositionTrack = composition.addMutableTrackWithMediaType(AVMediaTypeVideo, preferredTrackID: kCMPersistentTrackID_Invalid)
do{
try compositionTrack.insertTimeRange(track.timeRange, ofTrack: track, atTime: kCMTimeZero)
}
catch _{callback(url: nil); return}
let videoComposition = AVMutableVideoComposition(propertiesOfAsset: composition)
videoComposition.customVideoCompositorClass = VideoFilterCompositor.self
videoComposition.frameDuration = CMTimeMake(1, 30)
videoComposition.renderSize = compositionTrack.naturalSize
let instruction = VideoFilterCompositionInstruction(trackID: compositionTrack.trackID)
instruction.timeRange = CMTimeRangeMake(kCMTimeZero, self.asset.duration)
videoComposition.instructions = [instruction]
let session: AVAssetExportSession = AVAssetExportSession(asset: composition, presetName: AVAssetExportPresetMediumQuality)!
session.videoComposition = videoComposition
session.outputURL = url
session.outputFileType = AVFileTypeMPEG4
session.exportAsynchronouslyWithCompletionHandler(){
callback(url: url)
}
}
}
这是另外两个班级 - 我会将它们放在一个代码块中以缩短帖子
Here's the other two classes - I'll put them both into one code block to make this post shorter
// Video Filter Composition Instruction Class - from what I gather,
// AVVideoCompositionInstruction is used only to pass values to
// the AVVideoCompositing class
class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{
let trackID: CMPersistentTrackID
let filters: ImageFilterGroup
let context: CIContext
// When I leave this line as-is, startVideoCompositionRequest: isn't called.
// When commented out, startVideoCompositionRequest(request) is called, but there
// are no valid CVPixelBuffers provided by request.sourceFrameByTrackID(below value)
override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
override var requiredSourceTrackIDs: [NSValue]{get{return []}}
override var containsTweening: Bool{get{return false}}
init(trackID: CMPersistentTrackID, filters: ImageFilterGroup, context: CIContext){
self.trackID = trackID
self.filters = filters
self.context = context
super.init()
//self.timeRange = timeRange
self.enablePostProcessing = true
}
required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
}
// My custom AVVideoCompositing class. This is where the problem lies -
// although I don't know if this is the root of the problem
class VideoFilterCompositor : NSObject, AVVideoCompositing{
var requiredPixelBufferAttributesForRenderContext: [String : AnyObject] = [
kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA), // The video is in 32 BGRA
kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)
]
var sourcePixelBufferAttributes: [String : AnyObject]? = [
kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA),
kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)
]
let renderQueue = dispatch_queue_create("co.getblix.videofiltercompositor.renderingqueue", DISPATCH_QUEUE_SERIAL)
override init(){
super.init()
}
func startVideoCompositionRequest(request: AVAsynchronousVideoCompositionRequest){
// This code block is never executed when the
// passthroughTrackID variable is in the above class
autoreleasepool(){
dispatch_async(self.renderQueue){
guard let instruction = request.videoCompositionInstruction as? VideoFilterCompositionInstruction else{
request.finishWithError(NSError(domain: "getblix.co", code: 760, userInfo: nil))
return
}
guard let pixels = request.sourceFrameByTrackID(instruction.passthroughTrackID) else{
// This code block is executed when I comment out the
// passthroughTrackID variable in the above class
request.finishWithError(NSError(domain: "getblix.co", code: 761, userInfo: nil))
return
}
// I have not been able to get the code to reach this point
// This function is either not called, or the guard
// statement above executes
let image = CIImage(CVPixelBuffer: pixels)
let filtered: CIImage = //apply the filter here
let width = CVPixelBufferGetWidth(pixels)
let height = CVPixelBufferGetHeight(pixels)
let format = CVPixelBufferGetPixelFormatType(pixels)
var newBuffer: CVPixelBuffer?
CVPixelBufferCreate(kCFAllocatorDefault, width, height, format, nil, &newBuffer)
if let buffer = newBuffer{
instruction.context.render(filtered, toCVPixelBuffer: buffer)
request.finishWithComposedVideoFrame(buffer)
}
else{
request.finishWithComposedVideoFrame(pixels)
}
}
}
}
func renderContextChanged(newRenderContext: AVVideoCompositionRenderContext){
// I don't have any code in this block
}
// This is interesting - this is called twice,
// Once before startVideoCompositionRequest is called,
// And once after. In the case when startVideoCompositionRequest
// Is not called, this is simply called twice in a row
func cancelAllPendingVideoCompositionRequests(){
dispatch_barrier_async(self.renderQueue){
print("Cancelled")
}
}
}
我已经我一直在寻找 Apple的AVCustomEdit示例项目以获取相关指导，但我似乎无法找到为什么会发生这种情况。
I've been looking at Apple's AVCustomEdit sample project a lot for guidance with this, but I can't seem to find in it any reason why this is happening.
我怎样才能得到 request.sourceFrameByTrackID： 功能到正确调用，并为每个帧提供有效的 CVPixelBuffer ？
事实证明 requiredSourceTrackIDs 自定义变量 AVVideoCompositionInstruction class（ VideoFilterCompositionInstruction 必须设置为包含轨道ID的数组
It turns out that the requiredSourceTrackIDs variable in the custom AVVideoCompositionInstruction class (VideoFilterCompositionInstruction in the question) has to be set to an array containing the track IDs
override var requiredSourceTrackIDs: [NSValue]{
get{
return [
NSNumber(value: Int(self.trackID))
]
}
}
所以最终的自定义组合指令类是
So the final custom composition instruction class is
class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{
let trackID: CMPersistentTrackID
let filters: [CIFilter]
let context: CIContext
override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
override var requiredSourceTrackIDs: [NSValue]{get{return [NSNumber(value: Int(self.trackID))]}}
override var containsTweening: Bool{get{return false}}
init(trackID: CMPersistentTrackID, filters: [CIFilter], context: CIContext){
self.trackID = trackID
self.filters = filters
self.context = context
super.init()
self.enablePostProcessing = true
}
required init?(coder aDecoder: NSCoder){
fatalError("init(coder:) has not been implemented")
}
}
此实用程序的所有代码也是o n GitHub
All of the code for this utility is also on GitHub
这篇关于自定义AVVideoCompositing类无法按预期工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
