使用Android的MediaCodec和MediaMuxer合并AAC音频 [英] Muxing AAC audio with Android's MediaCodec and MediaMuxer
问题描述
I'm modifying an Android Framework example to package the elementary AAC streams produced by MediaCodec into a standalone .mp4 file. I'm using a single MediaMuxer instance containing one AAC track generated by a MediaCodec instance.
但是，我始终总是在呼叫mMediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo)时收到错误消息:
However I always eventually get an error message on a call to mMediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo):
E/MPEG4Writer﹕timestampUs 0 < lastTimestampUs XXXXX for Audio track
当我在mCodec.queueInputBuffer(...)中对原始输入数据进行排队时，根据框架示例，我将提供0作为时间戳值(我也尝试过使用单调递增的时间戳值来获得相同的结果.我已经成功地将原始摄像机帧编码为使用相同方法的h264/mp4文件).
When I queue the raw input data in mCodec.queueInputBuffer(...) I provide 0 as the timestamp value per the Framework Example (I've also tried using monotonically increasing timestamp values with the same result. I've successfully encoded raw camera frames to h264/mp4 files with this same method).
最相关的代码段:
private static void testEncoder(String componentName, MediaFormat format, Context c) {
int trackIndex = 0;
boolean mMuxerStarted = false;
File f = FileUtils.createTempFileInRootAppStorage(c, "aac_test_" + new Date().getTime() + ".mp4");
MediaCodec codec = MediaCodec.createByCodecName(componentName);
try {
codec.configure(
format,
null /* surface */,
null /* crypto */,
MediaCodec.CONFIGURE_FLAG_ENCODE);
} catch (IllegalStateException e) {
Log.e(TAG, "codec '" + componentName + "' failed configuration.");
}
codec.start();
try {
mMediaMuxer = new MediaMuxer(f.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
} catch (IOException ioe) {
throw new RuntimeException("MediaMuxer creation failed", ioe);
}
ByteBuffer[] codecInputBuffers = codec.getInputBuffers();
ByteBuffer[] codecOutputBuffers = codec.getOutputBuffers();
int numBytesSubmitted = 0;
boolean doneSubmittingInput = false;
int numBytesDequeued = 0;
while (true) {
int index;
if (!doneSubmittingInput) {
index = codec.dequeueInputBuffer(kTimeoutUs /* timeoutUs */);
if (index != MediaCodec.INFO_TRY_AGAIN_LATER) {
if (numBytesSubmitted >= kNumInputBytes) {
Log.i(TAG, "queueing EOS to inputBuffer");
codec.queueInputBuffer(
index,
0 /* offset */,
0 /* size */,
0 /* timeUs */,
MediaCodec.BUFFER_FLAG_END_OF_STREAM);
if (VERBOSE) {
Log.d(TAG, "queued input EOS.");
}
doneSubmittingInput = true;
} else {
int size = queueInputBuffer(
codec, codecInputBuffers, index);
numBytesSubmitted += size;
if (VERBOSE) {
Log.d(TAG, "queued " + size + " bytes of input data.");
}
}
}
}
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
index = codec.dequeueOutputBuffer(info, kTimeoutUs /* timeoutUs */);
if (index == MediaCodec.INFO_TRY_AGAIN_LATER) {
} else if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = codec.getOutputFormat();
trackIndex = mMediaMuxer.addTrack(newFormat);
mMediaMuxer.start();
mMuxerStarted = true;
} else if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
codecOutputBuffers = codec.getOutputBuffers();
} else {
// Write to muxer
ByteBuffer encodedData = codecOutputBuffers[index];
if (encodedData == null) {
throw new RuntimeException("encoderOutputBuffer " + index +
" was null");
}
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
// The codec config data was pulled out and fed to the muxer when we got
// the INFO_OUTPUT_FORMAT_CHANGED status. Ignore it.
if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
info.size = 0;
}
if (info.size != 0) {
if (!mMuxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
// adjust the ByteBuffer values to match BufferInfo (not needed?)
encodedData.position(info.offset);
encodedData.limit(info.offset + info.size);
mMediaMuxer.writeSampleData(trackIndex, encodedData, info);
if (VERBOSE) Log.d(TAG, "sent " + info.size + " audio bytes to muxer with pts " + info.presentationTimeUs);
}
codec.releaseOutputBuffer(index, false);
// End write to muxer
numBytesDequeued += info.size;
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
if (VERBOSE) {
Log.d(TAG, "dequeued output EOS.");
}
break;
}
if (VERBOSE) {
Log.d(TAG, "dequeued " + info.size + " bytes of output data.");
}
}
}
if (VERBOSE) {
Log.d(TAG, "queued a total of " + numBytesSubmitted + "bytes, "
+ "dequeued " + numBytesDequeued + " bytes.");
}
int sampleRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
int channelCount = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
int inBitrate = sampleRate * channelCount * 16; // bit/sec
int outBitrate = format.getInteger(MediaFormat.KEY_BIT_RATE);
float desiredRatio = (float)outBitrate / (float)inBitrate;
float actualRatio = (float)numBytesDequeued / (float)numBytesSubmitted;
if (actualRatio < 0.9 * desiredRatio || actualRatio > 1.1 * desiredRatio) {
Log.w(TAG, "desiredRatio = " + desiredRatio
+ ", actualRatio = " + actualRatio);
}
codec.release();
mMediaMuxer.stop();
mMediaMuxer.release();
codec = null;
}
更新:我发现了一个根源症状，我认为该症状位于MediaCodec之内.:
Update: I've found a root symptom I think lies within MediaCodec.:
我将presentationTimeUs=1000发送到queueInputBuffer(...)，但是在调用MediaCodec.dequeueOutputBuffer(info, timeoutUs)后收到info.presentationTimeUs= 33219. fadden留下了与此行为有关的有用评论.
I send presentationTimeUs=1000 to queueInputBuffer(...) but receive info.presentationTimeUs= 33219 after calling MediaCodec.dequeueOutputBuffer(info, timeoutUs). fadden left a helpful comment related to this behavior.
推荐答案
Thanks to fadden's help I've got a proof-of-concept audio encoder and video+audio encoder on Github. In summary:
将AudioRecord的样本发送到MediaCodec + MediaMuxer包装器.使用audioRecord.read(...)处的系统时间可以很好地用作音频时间戳，前提是您要进行足够的轮询以免填满AudioRecord的内部缓冲区(以避免在您调用读取的时间与AudioRecord记录样本的时间之间发生漂移).糟糕的AudioRecord无法直接传达时间戳...
Send AudioRecord's samples to a MediaCodec + MediaMuxer wrapper. Using the system time at audioRecord.read(...) works sufficiently well as an audio timestamp, provided you poll often enough to avoid filling up AudioRecord's internal buffer (to avoid drift between the time you call read and the time AudioRecord recorded the samples). Too bad AudioRecord doesn't directly communicate timestamps...
// Setup AudioRecord
while (isRecording) {
audioPresentationTimeNs = System.nanoTime();
audioRecord.read(dataBuffer, 0, samplesPerFrame);
hwEncoder.offerAudioEncoder(dataBuffer.clone(), audioPresentationTimeNs);
}
请注意，AudioRecord 仅保证支持16位PCM样本，尽管MediaCodec.queueInputBuffer将输入作为byte[].将byte[]传递给audioRecord.read(dataBuffer,...)将会 truncate 为您将16位样本拆分为8位.
Note that AudioRecord only guarantees support for 16 bit PCM samples, though MediaCodec.queueInputBuffer takes input as byte[]. Passing a byte[] to audioRecord.read(dataBuffer,...) will truncate split the 16 bit samples into 8 bit for you.
我发现以这种方式进行轮询仍然偶尔会产生timestampUs XXX < lastTimestampUs XXX for Audio track错误，因此我加入了一些逻辑来跟踪mediaCodec.dequeueOutputBuffer(bufferInfo, timeoutMs)报告的bufferInfo.presentationTimeUs，并在调用mediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo)之前进行必要的调整.
I found that polling in this way still occasionally generated a timestampUs XXX < lastTimestampUs XXX for Audio track error, so I included some logic to keep track of the bufferInfo.presentationTimeUs reported by mediaCodec.dequeueOutputBuffer(bufferInfo, timeoutMs) and adjust if necessary before calling mediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo).
这篇关于使用Android的MediaCodec和MediaMuxer合并AAC音频的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
