使用准确的CMTime将AudioBuffer转换为CMSampleBuffer [英] Converting AudioBuffer to CMSampleBuffer with accurate CMTime
问题描述
这里的目标是通过AVCaptureDataOutput通过视频创建一个mp4文件，并通过CoreAudio录制音频.然后将两者的CMSampleBuffers发送到具有AVAssetWriterInput(AVMediaTypeVideo)和AVAssetWriterInput(AVMediaTypeAudio)的AVAssetWriter
The goal here is to create a mp4 file via video through AVCaptureDataOutput and audio recorded a CoreAudio. Then send the CMSampleBuffers of both to an AVAssetWriter who has accompanying AVAssetWriterInput(AVMediaTypeVideo) and AVAssetWriterInput(AVMediaTypeAudio)
我的音频编码器将AudioBuffer复制到新的CMSampleBuffer，然后将其传递到AVAssetWriterInput(AVMediaTypeAudio).本示例说明了如何将AudioBuffer转换为CMSampleBuffer.转换为CMSampleBuffer
My audio encoder the copies the AudioBuffer to a new CMSampleBuffer the passes it to the AVAssetWriterInput(AVMediaTypeAudio). This example is how the conversion of AudioBuffer to CMSampleBuffer is done. Converstion to CMSampleBuffer
长话短说，这是行不通的.视频显示，但没有音频.
Long story short, it does not work. The video shows up but no audio.
但是，如果我注释掉了视频编码，那么音频将被写入文件并且可以听见.
BUT, if I comment out the video encoding, then the audio is written to the file and audible.
根据经验告诉我，这是一个时间问题.与CMSampleBuffer的对话确实显示了
That tells me from experience it is a timing issue. The Converstion to CMSampleBuffer does show
CMSampleTimingInfo timing = { CMTimeMake(1, 44100.0), kCMTimeZero, kCMTimeInvalid };
它产生的时间CMTimeCopyDescription为 {0/1 = 0.000} ，这对我来说似乎是完全错误的.我试图跟踪呈现的帧，并像这样传递时间值的帧数和时间比例的采样率
It produces a time CMTimeCopyDescription of {0/1 = 0.000} which seems completely wrong to me. I tried keeping track of the frames rendered and passing the framecount for the time value and the samplerate for the time scale like this
CMSampleTimingInfo timing = { CMTimeMake(1, 44100.0), CMTimeMake(self.frameCount, 44100.0), kCMTimeInvalid };
但是没有骰子.外观更漂亮的CMSampleTimingInfo {107520/44100 = 2.438} ，但文件中仍然没有音频.
But no dice. A nicer looking CMSampleTimingInfo {107520/44100 = 2.438}, but still no audio in the file.
视频CMSampleBuffer产生类似这样的 {65792640630624/1000000000 = 65792.641，四舍五入} .这告诉我AVCaptureVideoOutput的时间标度为10亿，可能为纳秒.我来宾，时间值就是设备时间.我找不到有关AVCaptureVideoOutput使用的任何信息.
The video CMSampleBuffer produces something like this {65792640630624/1000000000 = 65792.641, rounded}. This tells me the AVCaptureVideoOutput has a time scale of 1 billion, likely nanoseconds. And I guest the time value is the something like the device time. I cant find any info about what AVCaptureVideoOutput uses.
任何人都有有用的指导吗?我什至在正确的轨道上吗?
Anyone have any helpful guidance? Am I even on the right track?
在这里进行转化
CMSampleBufferRef buff = malloc(sizeof(CMSampleBufferRef));
CMFormatDescriptionRef format = NULL;
self.frameCount += inNumberFrames;
CMTime presentationTime = CMTimeMake(self.frameCount, self.pcmASBD.mSampleRate);
AudioStreamBasicDescription audioFormat = self.pcmASBD;
CheckError(CMAudioFormatDescriptionCreate(kCFAllocatorDefault,
&audioFormat,
0,
NULL,
0,
NULL,
NULL,
&format),
"Could not create format from AudioStreamBasicDescription");
CMSampleTimingInfo timing = { CMTimeMake(1, self.pcmASBD.mSampleRate), presentationTime, kCMTimeInvalid };
CheckError(CMSampleBufferCreate(kCFAllocatorDefault,
NULL,
false,
NULL,
NULL,
format,
(CMItemCount)inNumberFrames,
1,
&timing,
0,
NULL,
&buff),
"Could not create CMSampleBufferRef");
CheckError(CMSampleBufferSetDataBufferFromAudioBufferList(buff,
kCFAllocatorDefault,
kCFAllocatorDefault,
0,
audioBufferList),
"Could not set data in CMSampleBufferRef");
[self.delegate didRenderAudioSampleBuffer:buff];
CFRelease(buff);
还有我创建的assetWriters
And the assetWriters I create
func createVideoInputWriter()->AVAssetWriterInput? {
let numPixels = Int(self.size.width * self.size.height)
let bitsPerPixel:Int = 11
let bitRate = Int64(numPixels * bitsPerPixel)
let fps:Int = 30
let settings:[NSObject : AnyObject] = [
AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : self.size.width,
AVVideoHeightKey : self.size.height,
AVVideoCompressionPropertiesKey : [
AVVideoAverageBitRateKey : NSNumber(longLong: bitRate),
AVVideoMaxKeyFrameIntervalKey : NSNumber(integer: fps)
]
]
var assetWriter:AVAssetWriterInput!
if self.mainAssetWriter.canApplyOutputSettings(settings, forMediaType:AVMediaTypeVideo) {
assetWriter = AVAssetWriterInput(mediaType:AVMediaTypeVideo, outputSettings:settings)
assetWriter.expectsMediaDataInRealTime = true
if self.mainAssetWriter.canAddInput(assetWriter) {
self.mainAssetWriter.addInput(assetWriter)
}
}
return assetWriter;
}
func createAudioInputWriter()->AVAssetWriterInput? {
let settings:[NSObject : AnyObject] = [
AVFormatIDKey : kAudioFormatMPEG4AAC,
AVNumberOfChannelsKey : 2,
AVSampleRateKey : 44100,
AVEncoderBitRateKey : 64000
]
var assetWriter:AVAssetWriterInput!
if self.mainAssetWriter.canApplyOutputSettings(settings, forMediaType:AVMediaTypeAudio) {
assetWriter = AVAssetWriterInput(mediaType:AVMediaTypeAudio, outputSettings:settings)
assetWriter.expectsMediaDataInRealTime = true
if self.mainAssetWriter.canAddInput(assetWriter) {
self.mainAssetWriter.addInput(assetWriter)
} else {
let error = NSError(domain:CMHDFileEncoder.Domain, code:CMHDFileEncoderErrorCode.CantAddInput.rawValue, userInfo:nil)
self.errorDelegate.hdFileEncoderError(error)
}
} else {
let error = NSError(domain:CMHDFileEncoder.Domain, code:CMHDFileEncoderErrorCode.CantApplyOutputSettings.rawValue, userInfo:nil)
self.errorDelegate.hdFileEncoderError(error)
}
return assetWriter
}
推荐答案
当然，这个问题有两个星期了，在星期五晚上发布了问题，并在星期一早上找到了解决方案.
Of course, had the problem for 2 weeks, posted the question on a friday night, and found the solution monday morning.
我遇到的研究使我走上了正轨...
The research I came across with put me on the right track...
1000000000时标是纳秒.但是时间值必须是设备绝对时间的纳秒.
1000000000 timescale is for nano seconds. But the timevalue has to be nanoseconds of the devices absolute time.
这篇文章比我能更好地解释-马赫时间
This post explains better than I can - mach time
我最终使用此代码对其进行了修复
I ended up using this code to fix it
CMSampleBufferRef buff = malloc(sizeof(CMSampleBufferRef));
CMFormatDescriptionRef format = NULL;
AudioStreamBasicDescription audioFormat = self.pcmASBD;
CheckError(CMAudioFormatDescriptionCreate(kCFAllocatorDefault,
&audioFormat,
0,
NULL,
0,
NULL,
NULL,
&format),
"Could not create format from AudioStreamBasicDescription");
uint64_t time = inTimeStamp->mHostTime;
/* Convert to nanoseconds */
time *= info.numer;
time /= info.denom;
CMTime presentationTime = CMTimeMake(time, kDeviceTimeScale);
CMSampleTimingInfo timing = { CMTimeMake(1, self.pcmASBD.mSampleRate), presentationTime, kCMTimeInvalid };
CheckError(CMSampleBufferCreate(kCFAllocatorDefault,
NULL,
false,
NULL,
NULL,
format,
(CMItemCount)inNumberFrames,
1,
&timing,
0,
NULL,
&buff),
"Could not create CMSampleBufferRef");
CheckError(CMSampleBufferSetDataBufferFromAudioBufferList(buff,
kCFAllocatorDefault,
kCFAllocatorDefault,
0,
audioBufferList),
"Could not set data in CMSampleBufferRef");
这篇关于使用准确的CMTime将AudioBuffer转换为CMSampleBuffer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
