我可以使用AVCaptureSession将AAC流编码到内存中吗？ [英] Can I use AVCaptureSession to encode an AAC stream to memory?

问题描述

我正在编写一个通过网络传输视频和音频的iOS应用。

I'm writing an iOS app that streams video and audio over the network.

我正在使用 AVCaptureSession 来抓取原始视频帧使用 AVCaptureVideoDataOutput 并使用使用x264 对其进行编码。这很好用。

I am using AVCaptureSession to grab raw video frames using AVCaptureVideoDataOutput and encode them in software using x264. This works great.

我想对音频做同样的事情，只是因为我不需要在音频方面那么多控制所以我想使用内置的硬件编码器产生AAC流。这意味着使用音频转换器来自Audio Toolbox图层。为了做到这一点，我为 AVCaptudeAudioDataOutput 的音频帧添加了一个处理程序：

I wanted to do the same for audio, only that I don't need that much control on the audio side so I wanted to use the built in hardware encoder to produce an AAC stream. This meant using Audio Converter from the Audio Toolbox layer. In order to do so I put in a handler for AVCaptudeAudioDataOutput's audio frames:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection 
{
    // get the audio samples into a common buffer _pcmBuffer
    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &_pcmBufferSize, &_pcmBuffer);

    // use AudioConverter to
    UInt32 ouputPacketsCount = 1;
    AudioBufferList bufferList;
    bufferList.mNumberBuffers = 1;
    bufferList.mBuffers[0].mNumberChannels = 1;
    bufferList.mBuffers[0].mDataByteSize = sizeof(_aacBuffer);
    bufferList.mBuffers[0].mData = _aacBuffer;
    OSStatus st = AudioConverterFillComplexBuffer(_converter, converter_callback, (__bridge void *) self, &ouputPacketsCount, &bufferList, NULL);
    if (0 == st) {
        // ... send bufferList.mBuffers[0].mDataByteSize bytes from _aacBuffer...
    }
}

在这种情况下，音频转换器的回调函数非常简单（假设数据包大小和计数设置正确）：

In this case the callback function for the audio converter is pretty simple (assuming packet sizes and counts are setup properly):

- (void) putPcmSamplesInBufferList:(AudioBufferList *)bufferList withCount:(UInt32 *)count
{
    bufferList->mBuffers[0].mData = _pcmBuffer;         
    bufferList->mBuffers[0].mDataByteSize = _pcmBufferSize;
}

音频转换器的设置如下所示：

And the setup for the audio converter looks like this:

{
    // ...
    AudioStreamBasicDescription pcmASBD = {0};
    pcmASBD.mSampleRate = ((AVAudioSession *) [AVAudioSession sharedInstance]).currentHardwareSampleRate;
    pcmASBD.mFormatID = kAudioFormatLinearPCM;
    pcmASBD.mFormatFlags = kAudioFormatFlagsCanonical;
    pcmASBD.mChannelsPerFrame = 1;
    pcmASBD.mBytesPerFrame = sizeof(AudioSampleType);
    pcmASBD.mFramesPerPacket = 1;
    pcmASBD.mBytesPerPacket = pcmASBD.mBytesPerFrame * pcmASBD.mFramesPerPacket;
    pcmASBD.mBitsPerChannel = 8 * pcmASBD.mBytesPerFrame;

    AudioStreamBasicDescription aacASBD = {0};
    aacASBD.mFormatID = kAudioFormatMPEG4AAC;
    aacASBD.mSampleRate = pcmASBD.mSampleRate;
    aacASBD.mChannelsPerFrame = pcmASBD.mChannelsPerFrame;
    size = sizeof(aacASBD);
    AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &aacASBD);

    AudioConverterNew(&pcmASBD, &aacASBD, &_converter);
    // ...
}

这似乎很简单它不工作。 AVCaptureSession运行后，音频转换器（特别是AudioConverterFillComplexBuffer）返回'hwiu'（硬件使用中）错误。如果会话停止但是我无法捕获任何内容，转换工作正常......

This seems pretty straight forward only the IT DOES NOT WORK. Once the AVCaptureSession is running, the audio converter (specifically AudioConverterFillComplexBuffer) returns an 'hwiu' (hardware in use) error. Conversion works fine if the session is stopped but then I can't capture anything...

我想知道是否有办法从AVCaptureSession中获取AAC流。我正在考虑的选项是：

I was wondering if there was a way to get an AAC stream out of AVCaptureSession. The options I'm considering are:

1. 以某种方式使用AVAssetWriterInput将音频样本编码到AAC中，然后以某种方式获取编码数据包（不通过AVAssetWriter，它只会写入文件）。

1. Somehow using AVAssetWriterInput to encode audio samples into AAC and then get the encoded packets somehow (not through AVAssetWriter, which would only write to a file).

重组我的应用程序，使其仅在视频端使用AVCaptureSession并使用音频队列。这将使流控制（开始和停止录制，响应中断）变得更加复杂，我担心它可能会导致音频和视频之间的同步问题。此外，它似乎不是一个好的设计。

Reorganizing my app so that it uses AVCaptureSession only on the video side and uses Audio Queues on the audio side. This will make flow control (starting and stopping recording, responding to interruptions) more complicated and I'm afraid that it might cause synching problems between the audio and video. Also, it just doesn't seem like a good design.

有没有人知道是否从AVCaptureSession获取AAC有可能吗？我必须在这里使用音频队列吗？这会让我进入同步或控制问题吗？

Does anyone know if getting the AAC out of AVCaptureSession is possible? Do I have to use Audio Queues here? Could this get me into synching or control problems?

推荐答案

我最后向Apple寻求建议（事实证明你可以做到这一点如果您有付费开发者帐户。）

I ended up asking Apple for advice (it turns out you can do that if you have a paid developer account).

AVCaptureSession似乎抓住了AAC硬件编码器，但只允许您使用它直接写入文件。

It seems that AVCaptureSession grabs a hold of the AAC hardware encoder but only lets you use it to write directly to file.

您可以使用软件编码器，但您必须专门询问它而不是使用AudioConverterNew：

You can use the software encoder but you have to ask for it specifically instead of using AudioConverterNew:

AudioClassDescription *description = [self
        getAudioClassDescriptionWithType:kAudioFormatMPEG4AAC
                        fromManufacturer:kAppleSoftwareAudioCodecManufacturer];
if (!description) {
    return false;
}
// see the question as for setting up pcmASBD and arc ASBD
OSStatus st = AudioConverterNewSpecific(&pcmASBD, &aacASBD, 1, description, &_converter);
if (st) {
    NSLog(@"error creating audio converter: %s", OSSTATUS(st));
    return false;
}

- (AudioClassDescription *)getAudioClassDescriptionWithType:(UInt32)type
                                           fromManufacturer:(UInt32)manufacturer
{
    static AudioClassDescription desc;

    UInt32 encoderSpecifier = type;
    OSStatus st;

    UInt32 size;
    st = AudioFormatGetPropertyInfo(kAudioFormatProperty_Encoders,
                                    sizeof(encoderSpecifier),
                                    &encoderSpecifier,
                                    &size);
    if (st) {
        NSLog(@"error getting audio format propery info: %s", OSSTATUS(st));
        return nil;
    }

    unsigned int count = size / sizeof(AudioClassDescription);
    AudioClassDescription descriptions[count];
    st = AudioFormatGetProperty(kAudioFormatProperty_Encoders,
                                sizeof(encoderSpecifier),
                                &encoderSpecifier,
                                &size,
                                descriptions);
    if (st) {
        NSLog(@"error getting audio format propery: %s", OSSTATUS(st));
        return nil;
    }

    for (unsigned int i = 0; i < count; i++) {
        if ((type == descriptions[i].mSubType) &&
            (manufacturer == descriptions[i].mManufacturer)) {
            memcpy(&desc, &(descriptions[i]), sizeof(desc));
            return &desc;
        }
    }

    return nil;
}

软件编码器当然会占用CPU资源，但会获得完成工作。

The software encoder will take up CPU resources, of course, but will get the job done.

这篇关于我可以使用AVCaptureSession将AAC流编码到内存中吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！