如何控制AVAssetWriter以正确的FPS写入 [英] How do I control AVAssetWriter to write at the correct FPS
问题描述
让我看看我是否理解正确。
在目前最先进的硬件上，iOS允许我以下列fps记录：30,60,120和240.
但这些fps表现不同。如果我以30或60 fps的速度拍摄，我希望以这些fps拍摄的视频文件分别以30和60 fps的速度播放。
但如果我在120拍摄或者240 fps，我希望以这些fps拍摄的视频文件以30 fps的速度播放，否则我将看不到慢动作。
几个问题：
- 我是对的吗？
- 有没有办法以120或240 fps的速度拍摄并在分别为120和240 fps？我的意思是在fps播放视频时没有慢慢拍摄？
- 我在编写文件时如何控制该帧速率？
我正在创建这样的AVAssetWriter输入...
NSDictionary * videoCompressionSettings = @ {AVVideoCodecKey：AVVideoCodecH264，
AVVideoWidthKey：@（videoWidth），
AVVideoHeightKey：@（videoHeight），
AVVideoCompressionPropertiesKey：@ {AVVideoAverageBitRateKey：@（bitsPerSecond），
AVVideoMaxKeyFrameIntervalKey：@ （1）}
};
_assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType：AVMediaTypeVideo outputSettings：videoCompressionSettings];
并且没有明显的方法来控制它。
注意：我尝试过不同的数字 1 。我试过 1.0 / fps ，我试过 fps ，我已经删除了密钥。没有区别。
这就是我设置`AVAssetWriter的方式：
AVAssetWriter * newAssetWriter = [[AVAssetWriter alloc] initWithURL：_movieURL fileType：AVFileTypeQuickTimeMovie
error：& error];
_assetWriter = newAssetWriter;
_assetWriter.shouldOptimizeForNetworkUse = NO;
CGFloat videoWidth = size.width;
CGFloat videoHeight = size.height;
NSUInteger numPixels = videoWidth * videoHeight;
NSUInteger bitsPerSecond;
//假设低于SD的分辨率用于流式传输，并使用较低的比特率
// if（numPixels<（640 * 480））
/ / bitsPerPixel = 4.05; //此比特率与AVCaptureSessionPresetMedium或Low产生的质量相匹配。
// else
NSUInteger bitsPerPixel = 11.4; //此比特率与AVCaptureSessionPresetHigh产生的质量相匹配。
bitsPerSecond = numPixels * bitsPerPixel;
NSDictionary * videoCompressionSettings = @ {AVVideoCodecKey：AVVideoCodecH264，
AVVideoWidthKey：@（videoWidth），
AVVideoHeightKey：@（videoHeight），
AVVideoCompressionPropertiesKey：@ {AVVideoAverageBitRateKey： @（bitsPerSecond）}
};
if（！[_ assetWriter canApplyOutputSettings：videoCompressionSettings forMediaType：AVMediaTypeVideo]）{
NSLog（@无法添加资产编写者视频输入。）;
返回;
}
_assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType：AVMediaTypeVideo
outputSettings：videoCompressionSettings
sourceFormatHint：formatDescription];
_assetWriterVideoInput.expectsMediaDataInRealTime = YES;
NSDictionary * adaptorDict = @ {
（id）kCVPixelBufferPixelFormatTypeKey：@（kCVPixelFormatType_32BGRA），
（id）kCVPixelBufferWidthKey：@（videoWidth），
（id）kCVPixelBufferHeightKey ：@（videoHeight）
};
_pixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput：_assetWriterVideoInput
sourcePixelBufferAttributes：adaptorDict];
//将资产编写者输入添加到资产编写者
if（！[_ assetWriter canAddInput：_assetWriterVideoInput]）{
return;
}
[_assetWriter addInput：_assetWriterVideoInput];
captureOutput 方法非常简单。我从过滤器获取图像并使用以下方法将其写入文件：
if（videoJustStartWriting）
[_assetWriter startSessionAtSourceTime： presentationTime];
CVPixelBufferRef renderedOutputPixelBuffer = NULL;
OSStatus err = CVPixelBufferPoolCreatePixelBuffer（nil，
_pixelBufferAdaptor.pixelBufferPool，
& renderedOutputPixelBuffer）;
if（err）return; // NSLog（@无法从缓冲池中获取像素缓冲区）;
// _ ciContext是一个金属上下文
[_ciContext render：finalImage
toCVPixelBuffer：renderedOutputPixelBuffer
bounds：[finalImage extent]
colorSpace：_sDeviceRgbColorSpace];
[self writeVideoPixelBuffer：renderedOutputPixelBuffer
withInitialTime：presentationTime];
- （void）writeVideoPixelBuffer：（CVPixelBufferRef）pixelBuffer withInitialTime：（CMTime）presentationTime
{
if（_assetWriter.status == AVAssetWriterStatusUnknown） {
//如果资产编写者状态未知，则意味着编写尚未开始，因此开始使用开始时间作为缓冲区的演示时间戳写入
if（[_assetWriter startWriting]）{
[_assetWriter startSessionAtSourceTime：presentationTime];
}
}
if（_assetWriter.status == AVAssetWriterStatusWriting）{
//如果资产编写者状态正在写入，请将样本缓冲区附加到其对应的资产编写者输入
if（_assetWriterVideoInput.readyForMoreMediaData）{
if（！[_ pixelBufferAdaptor appendPixelBuffer：pixelBuffer withPresentationTime：presentationTime]）{
NSLog（@error，[_ assetWriter.error localizedFailureReason ]）;
}
}
}
if（_assetWriter.status == AVAssetWriterStatusFailed）{
NSLog（@failed）;
}
}
我把整个事情都拍了在240帧/秒。这些是附加帧的呈现时间。
time ======= 113594.311510508
time === ==== 113594.324011508
time ======= 113594.328178716
time ======= 113594.340679424
time ======= 113594.344846383
如果你在它们之间进行一些计算，你会发现帧速率约为240 fps。因此帧的存储时间正确。
但是当我观看视频时，移动不是慢动作，快速时间表示视频是30 fps。 / p>
注意：此应用程序从相机抓取帧，帧进入CIF过滤器，这些过滤器的结果将转换回存储到文件并显示在其上的样本缓冲区屏幕。
我到达这里，但我认为这是你出错的地方。将您的视频捕获视为管道。
（1）捕获缓冲区 - > （2）用缓冲区做某事 - > （3）将缓冲区写为视频中的帧。
听起来你已经成功完成了（1）和（2），你得到了缓冲区足够快，你正在处理它们，所以你可以把它们作为帧出售。
问题几乎可以肯定是（3）写视频帧。
https://developer.apple .com / reference / avfoundation / avmutablevideocomposition
检查AVMutableComposition中的frameDuration设置，你需要像CMTime（1,60）这样的东西// 60FPS或CMTime（1,240）// 240FPS得到你想要的东西（告诉视频写这么多帧并以这个速率编码）。
使用AVAssetWriter，它的原理完全相同，但您将帧速率设置为AVAsetWriterInput outputSettings中添加AVVideoExpectedSourceFrameRateKey的属性。
NSDictionary * videoCompressionSettings = @ {AVVideoCodecKey：AVVideoCodecH264，
AVVideoWidthKey：@（videoWidth），
AVVideoHeightKey：@（videoHeight），
AVVideoExpectedSourceFrameRateKey：@（60），
AVVideoCompressionPropertiesKey：@ {AVVideoAverageBitRateKey：@（bitsPerSecond），
AVVideoMaxKeyFrameIntervalKey：@（1）}
};
要扩展一点 - 你无法严格控制或同步你的摄像头捕获到输出/播放速率，时间只是不起作用，并不是那么精确，当然处理管道增加了开销。当您捕获帧时，它们会被标记为时间戳，但是在写入/压缩阶段，它只使用它所需的帧来生成为合成指定的输出。
它有两种方式，你只能捕获30 FPS并以240 FPS写出，视频显示正常，你只需要很多帧丢失并被算法填充。你甚至可以每秒仅售出1帧并以30FPS回放，两者相互分离（我捕获的速度有多快以及我每秒出现的帧数）
至于如何以不同的速度播放，你只需要调整播放速度 - 根据需要减慢播放速度。
如果你设置正确时基（frameDuration），它将始终播放正常 - 你告诉它回放是每秒X帧，当然，你的眼睛可能会注意到差异（几乎可以肯定在低FPS和高FPS之间） ，屏幕可能无法刷新到那么高（60FPS以上），但无论视频的时基是多少都是正常的1倍速。通过减慢视频速度，如果我的时基为120，我将其减慢到.5x，我知道有效地看到60FPS，一秒钟的播放需要两秒钟。
你控制了通过在AVPlayer上设置rate属性来播放速度 https://developer.apple.com/reference/avfoundation / avplayer
Let me see if I understood it correctly.
At the present most advanced hardware, iOS allows me to record at the following fps: 30, 60, 120 and 240.
But these fps behave differently. If I shoot at 30 or 60 fps, I expect the videos files created from shooting at these fps to play at 30 and 60 fps respectively.
But if I shoot at 120 or 240 fps, I expect the video files creating from shooting at these fps to play at 30 fps, or I will not see the slow motion.
A few questions:
- am I right?
- is there a way to shoot at 120 or 240 fps and play at 120 and 240 fps respectively? I mean play at the fps the videos were shoot without slo-mo?
- How do I control that framerate when I write the file?
I am creating the AVAssetWriter input like this...
NSDictionary *videoCompressionSettings = @{AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : @(videoWidth),
AVVideoHeightKey : @(videoHeight),
AVVideoCompressionPropertiesKey : @{ AVVideoAverageBitRateKey : @(bitsPerSecond),
AVVideoMaxKeyFrameIntervalKey : @(1)}
};
_assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoCompressionSettings];
and there is no apparent way to control that.
NOTE: I have tried different numbers where that 1 is. I have tried 1.0/fps, I have tried fps and I have removed the key. No difference.
This is how I setup `AVAssetWriter:
AVAssetWriter *newAssetWriter = [[AVAssetWriter alloc] initWithURL:_movieURL fileType:AVFileTypeQuickTimeMovie
error:&error];
_assetWriter = newAssetWriter;
_assetWriter.shouldOptimizeForNetworkUse = NO;
CGFloat videoWidth = size.width;
CGFloat videoHeight = size.height;
NSUInteger numPixels = videoWidth * videoHeight;
NSUInteger bitsPerSecond;
// Assume that lower-than-SD resolutions are intended for streaming, and use a lower bitrate
// if ( numPixels < (640 * 480) )
// bitsPerPixel = 4.05; // This bitrate matches the quality produced by AVCaptureSessionPresetMedium or Low.
// else
NSUInteger bitsPerPixel = 11.4; // This bitrate matches the quality produced by AVCaptureSessionPresetHigh.
bitsPerSecond = numPixels * bitsPerPixel;
NSDictionary *videoCompressionSettings = @{AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : @(videoWidth),
AVVideoHeightKey : @(videoHeight),
AVVideoCompressionPropertiesKey : @{ AVVideoAverageBitRateKey : @(bitsPerSecond)}
};
if (![_assetWriter canApplyOutputSettings:videoCompressionSettings forMediaType:AVMediaTypeVideo]) {
NSLog(@"Couldn't add asset writer video input.");
return;
}
_assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoCompressionSettings
sourceFormatHint:formatDescription];
_assetWriterVideoInput.expectsMediaDataInRealTime = YES;
NSDictionary *adaptorDict = @{
(id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA),
(id)kCVPixelBufferWidthKey : @(videoWidth),
(id)kCVPixelBufferHeightKey : @(videoHeight)
};
_pixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput:_assetWriterVideoInput
sourcePixelBufferAttributes:adaptorDict];
// Add asset writer input to asset writer
if (![_assetWriter canAddInput:_assetWriterVideoInput]) {
return;
}
[_assetWriter addInput:_assetWriterVideoInput];
captureOutput method is very simple. I get the image from the filter and write it to file using:
if (videoJustStartWriting)
[_assetWriter startSessionAtSourceTime:presentationTime];
CVPixelBufferRef renderedOutputPixelBuffer = NULL;
OSStatus err = CVPixelBufferPoolCreatePixelBuffer(nil,
_pixelBufferAdaptor.pixelBufferPool,
&renderedOutputPixelBuffer);
if (err) return; // NSLog(@"Cannot obtain a pixel buffer from the buffer pool");
//_ciContext is a metal context
[_ciContext render:finalImage
toCVPixelBuffer:renderedOutputPixelBuffer
bounds:[finalImage extent]
colorSpace:_sDeviceRgbColorSpace];
[self writeVideoPixelBuffer:renderedOutputPixelBuffer
withInitialTime:presentationTime];
- (void)writeVideoPixelBuffer:(CVPixelBufferRef)pixelBuffer withInitialTime:(CMTime)presentationTime
{
if ( _assetWriter.status == AVAssetWriterStatusUnknown ) {
// If the asset writer status is unknown, implies writing hasn't started yet, hence start writing with start time as the buffer's presentation timestamp
if ([_assetWriter startWriting]) {
[_assetWriter startSessionAtSourceTime:presentationTime];
}
}
if ( _assetWriter.status == AVAssetWriterStatusWriting ) {
// If the asset writer status is writing, append sample buffer to its corresponding asset writer input
if (_assetWriterVideoInput.readyForMoreMediaData) {
if (![_pixelBufferAdaptor appendPixelBuffer:pixelBuffer withPresentationTime:presentationTime]) {
NSLog(@"error", [_assetWriter.error localizedFailureReason]);
}
}
}
if ( _assetWriter.status == AVAssetWriterStatusFailed ) {
NSLog(@"failed");
}
}
I put the whole thing to shoot at 240 fps. These are presentation times of frames being appended.
time ======= 113594.311510508
time ======= 113594.324011508
time ======= 113594.328178716
time ======= 113594.340679424
time ======= 113594.344846383
if you do some calculation between them you will see that the framerate is about 240 fps. So the frames are being stored with the correct time.
But when I watch the video the movement is not in slow motion and quick time says the video is 30 fps.
Note: this app grabs frames from the camera, the frames goes into CIFilters and the result of those filters is converted back to a sample buffer that is stored to file and displayed on the screen.
I'm reaching here, but I think this is where you're going wrong. Think of your video capture as a pipeline.
(1) Capture buffer -> (2) Do Something With buffer -> (3) Write buffer as frames in video.
Sounds like you've successfully completed (1) and (2), you're getting the buffer fast enough and you're processing them so you can vend them as frames.
The problem is almost certainly in (3) writing the video frames.
https://developer.apple.com/reference/avfoundation/avmutablevideocomposition
Check out the frameDuration setting in your AVMutableComposition, you'll need something like CMTime(1, 60) //60FPS or CMTime(1, 240) // 240FPS to get what you're after (telling the video to WRITE this many frames and encode at this rate).
Using AVAssetWriter, it's exactly the same principle but you set the frame rate as a property in the AVAssetWriterInput outputSettings adding in the AVVideoExpectedSourceFrameRateKey.
NSDictionary *videoCompressionSettings = @{AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : @(videoWidth),
AVVideoHeightKey : @(videoHeight),
AVVideoExpectedSourceFrameRateKey : @(60),
AVVideoCompressionPropertiesKey : @{ AVVideoAverageBitRateKey : @(bitsPerSecond),
AVVideoMaxKeyFrameIntervalKey : @(1)}
};
To expand a little more - you can't strictly control or sync your camera capture exactly to the output / playback rate, the timing just doesn't work that way and isn't that exact, and of course the processing pipeline adds overhead. When you capture frames they are time stamped, which you've seen, but in the writing / compression phase, it's using only the frames it needs to produce the output specified for the composition.
It goes both ways, you could capture only 30 FPS and write out at 240 FPS, the video would display fine, you'd just have a lot of frames "missing" and being filled in by the algorithm. You can even vend only 1 frame per second and play back at 30FPS, the two are separate from each other (how fast I capture Vs how many frames and what I present per second)
As to how to play it back at different speed, you just need to tweak the playback speed - slow it down as needed.
If you've correctly set the time base (frameDuration), it will always play back "normal" - you're telling it "play back is X Frames Per Second", of course, your eye may notice a difference (almost certainly between low FPS and high FPS), and the screen may not refresh that high (above 60FPS), but regardless the video will be at a "normal" 1X speed for it's timebase. By slowing the video, if my timebase is 120, and I slow it to .5x I know effectively see 60FPS and one second of playback takes two seconds.
You control the playback speed by setting the rate property on AVPlayer https://developer.apple.com/reference/avfoundation/avplayer
这篇关于如何控制AVAssetWriter以正确的FPS写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
