如何从MediaCodec解码器的输出中提取PCM样本 [英] How to extract PCM samples from MediaCodec decoder's output

问题描述

我正在尝试从解码的mp4缓冲区中获取PCM样本以进行进一步处理.我首先要从用手机的相机应用程序录制的视频文件中提取音轨，并且确保在我获得"audio/mp4" mime键时已选择音轨:

I'm trying to obtain the PCM samples for further processing from a decoded mp4 buffer. I'm first extracting the audio track from a video file recorded with the phone's camera app, and I've made sure the audio track is being selected when I get the 'audio/mp4' mime key:

MediaExtractor extractor = new MediaExtractor();
try {
    extractor.setDataSource(fileUri.getPath());
} catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}
int numTracks = extractor.getTrackCount();
for(int i =0; i<numTracks; ++i) {
    MediaFormat format = extractor.getTrackFormat(i);
    String mime = format.getString(MediaFormat.KEY_MIME);
    //Log.d("mime =",mime);
    if(mime.startsWith("audio/")) {
        extractor.selectTrack(i);
        decoder = MediaCodec.createDecoderByType(mime);
        decoder.configure(format, null, null, 0);

        //getSampleCryptoInfo(MediaCodec.CryptoInfo info)
        break;
    }
}
if (decoder == null) {
    Log.e("DecodeActivity", "Can't find audio info!");
    return;
}
decoder.start();

此后，我遍历轨道，向编解码器提供编码后的访问单元流，然后将解码后的访问单元拉入ByteBuffer(这是我从此处发布的视频渲染示例中回收的代码

After that, I iterate through the track, feeding the codec the stream of encoded access units, and pulling the decoded access units into a ByteBuffer (this is code I recycled from a video rendering example posted here https://github.com/vecio/MediaCodecDemo):

ByteBuffer[] inputBuffers = decoder.getInputBuffers();
ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
BufferInfo info = new BufferInfo();

boolean isEOS = false;

while (true) {
    if (!isEOS) {
        int inIndex = decoder.dequeueInputBuffer(10000);
        if (inIndex >= 0) {
            ByteBuffer buffer = inputBuffers[inIndex];
            int sampleSize = extractor.readSampleData(buffer, 0);
            if (sampleSize < 0) {
                // We shouldn't stop the playback at this point, just pass the EOS
                // flag to decoder, we will get it again from the
                // dequeueOutputBuffer
                Log.d("DecodeActivity", "InputBuffer BUFFER_FLAG_END_OF_STREAM");
                decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                isEOS = true;
            } else {
                decoder.queueInputBuffer(inIndex, 0, sampleSize, extractor.getSampleTime(), 0);
                extractor.advance();
            }
        }
    }

    int outIndex = decoder.dequeueOutputBuffer(info, 10000);
    switch (outIndex) {
    case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
        Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
        outputBuffers = decoder.getOutputBuffers();
        break;
    case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
        Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());
        break;
    case MediaCodec.INFO_TRY_AGAIN_LATER:
        Log.d("DecodeActivity", "dequeueOutputBuffer timed out!");
        break;
    default:
        ByteBuffer buffer = outputBuffers[outIndex];
        // How to obtain PCM samples from this buffer variable??

        decoder.releaseOutputBuffer(outIndex, true);
        break;
    }

    // All decoded frames have been rendered, we can stop playing now
    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
        Log.d("DecodeActivity", "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
        break;
    }
}

到目前为止，该代码似乎没有任何错误，但是我目前仍在尝试找出如何从采用输出缓冲区值的ByteBuffer中获取PCM样本.我猜想我可以假设，因为我正在使用16位立体声音频文件，所以在交错方案中应该至少有两个字节...但是我不太确定是否要这样做，因此可以毫无疑问地检索PCM样本.从此字节流.有人知道如何从MediaCodec API中获取这些信息吗?

The code seems to work with no errors so far, but I'm currently stuck at trying to figure out how to obtain the PCM samples from the ByteBuffer that is taking the value of the output buffer. I guess I could assume that since I'm working with 16-bit stereo audio file, there should be at least two bytes in an interleaved scheme... however I'm not really sure abut this, so to unequivocally retrieve the PCM samples from this byte stream. Does anybody know how get these from the MediaCodec API?

我已经阅读了一些使用ffmpeg或openSL的替代方法，但是由于我是Android编程的新手，所以我希望避免使用基于C的API的麻烦，并且仅使用Android提供的工具来构建我的第一个应用程序框架(我正在使用KitKat).任何帮助将不胜感激.

I've read a couple of alternatives using ffmpeg or openSL, but since I am new to Android programming I was hoping to avoid the complications of using c-based APIs and build my first app using only the tools provided by the Android Framework (I'm using KitKat). Any help will be greatly appreciated.

更新:我能够提取PCM样本，这是我假设的提取方式，也是way @ marcone指出的方式.为此，我在缓冲区分配下面添加了以下几行:

UPDATE: I was able to extract the PCM samples, the way I was assuming to do it and also the way@marcone pointed out. To do so, I added these lines below the buffer assignment:

byte[] b = new byte[info.size-info.offset];                         
int a = buffer.position();
buffer.get(b);
buffer.position(a);

最后通过以下方式将字节数组写入文件:

and finally write the byte array to a file by:

f.write(b,0,info.size-info.offset);

我现在要解决的问题是:

The problem I'm dealing now with is:

解码的音频样本与iZotope对mp4音轨的解码不完全匹配.波形文件大小中有48个样本不匹配，解码信号中有2112个样本延迟.我现在的问题是:所有mp4解码器会产生相同的输出PCM流，还是取决于解码器的实现?

The decoded audio samples do not exactly match with the decoding of the mp4 audio track done by iZotope. there is a 48 samples mismatch in the wave files size, and a 2112 samples delay in the decoded signals. My question now is: would all the mp4 decoders yield the same output PCM stream, or is it dependent on the implementation of the decoder?

推荐答案

我发现延迟是由AAC编码启动和剩余时间引起的，如此处所述:

I found the delays to be caused by the AAC encoding priming and remainder times, as explained here:

https://developer.apple.com/library/mac/documentation/quicktime/qtff/QTFFAppenG/QTFFAppenG.html

在我的情况下，启动时间始终为2112个样本，其余时间自然取决于音频大小而变化.

In my case, the priming time is always 2112 samples, and the remainder is naturally variable depending on the audio size.

这篇关于如何从MediaCodec解码器的输出中提取PCM样本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！