如何正确处理PTS采用了Android AudioRecord和媒体codeC音频连接codeR？ [英] How to handle the PTS correctly using Android AudioRecord and MediaCodec as audio encoder?

问题描述

我使用AudioRecord记录摄像机的Andr​​oid设备上捕获过程中音频流。
因为我要处理的帧数据和处理音频/视频样本，我不使用MediaRecorder。

我在读的调用另一个线程运行AudioRecord（）来收集原始音频数据。
一旦我得到的数据流，我喂它们到配置为AAC音频连接codeR的媒体codeC。

下面是我的一些codeS有关音频录音机/ EN codeR：

  m_en code_audio_mime =音频/ MP4A-LATM
m_audio_sample_rate = 44100;
m_audio_channels = AudioFormat.CHANNEL_IN_MONO;
m_audio_channel_count =（m_audio_channels == AudioFormat.CHANNEL_IN_MONO 1：2）;INT audio_bit_rate = 64000;
INT audio_data_format = AudioFormat.ENCODING_PCM_16BIT;m_audio_buffer_size = AudioRecord.getMinBufferSize（m_audio_sample_rate，m_audio_channels，audio_data_format）* 2;
m_audio_recorder =新AudioRecord（MediaRecorder.AudioSource.MIC，m_audio_sample_rate，
                                   m_audio_channels，audio_data_format，m_audio_buffer_size）;m_audio_en codeR =媒体codec.createEn coderByType（m_en code_audio_mime）;
MediaFormat audio_format =新MediaFormat（）;
audio_format.setString（MediaFormat.KEY_MIME，m_en code_audio_mime）;
audio_format.setInteger（MediaFormat.KEY_BIT_RATE，audio_bit_rate）;
audio_format.setInteger（MediaFormat.KEY_CHANNEL_COUNT，m_audio_channel_count）;
audio_format.setInteger（MediaFormat.KEY_SAMPLE_RATE，m_audio_sample_rate）;
audio_format.setInteger（MediaFormat.KEY_AAC_PROFILE，媒体codecInfo codecProfileLevel.AACObjectLC）;
audio_format.setInteger（MediaFormat.KEY_MAX_INPUT_SIZE，m_audio_buffer_size）;
m_audio_en coder.configure（audio_format，NULL，NULL，媒体codec.CONFIGURE_FLAG_EN code）;
 

我发现的 AudioRecord.read的第一次（）需要更长的时间来恢复，而连续的读取（）有更接近的音频数据的实时时间间隔。
例如，我的音频格式是44100Hz 16位1路，和AudioRecord的缓冲区大小为16384，所以一个完整的缓冲装置185.76毫秒。当我记录系统时间读的每次调用（），并从基本时间减去他们，我得到以下顺序：

每个时间之前阅读（）：0毫秒，345ms，543ms，692ms，891ms，1093ms，1244ms，...

我喂这些原始资料，音频带上面的时间值PTS codeR和EN codeR输出连接codeD音频样本以下PTS：

连接codeR输出PTS：0毫秒，185ms，371ms，557ms，743ms，928ms，...

它看起来像的EN codeR对待数据的每个部分具有相同的时间段。我相信EN codeR正常工作，因为我给它具有相同的大小（16384）每次原始数据。不过，如果我使用的是EN codeR输出作为PTS的复用器输入，我会用音频内容的速度更快，然后视频内容的视频。

我要问的是：

1. 是不是预期AudioRecord.read（）块在第一时间更长？我敢肯定，在函数调用花费的时间超过300毫秒，而它只记录16384字节为186ms。这还取决于设备/ Android版本的问题？


2. 我应该怎么做才能实现音频/视频同步？我有一个变通方法来测量读取（第一次调用的延迟时间），然后通过延迟切换音频采样的PTS。是否有另一种更好的方式来处理呢？

解决方案

转换单声道输入立体声。我拉我的头发前一段时间我才意识到AAC连接由媒体codeR暴露codeR只有立体声输入工作。

I'm using AudioRecord to record the audio stream during a camera capturing process on Android device. Since I want to process the frame data and handle audio/video samples, I do not use MediaRecorder.

I run AudioRecord in another thread with the calling of read() to gather the raw audio data. Once I get a data stream, I feed them into an MediaCodec configured as an AAC audio encoder.

Here are some of my codes about the audio recorder / encoder:

m_encode_audio_mime = "audio/mp4a-latm";
m_audio_sample_rate = 44100;
m_audio_channels = AudioFormat.CHANNEL_IN_MONO;
m_audio_channel_count = (m_audio_channels == AudioFormat.CHANNEL_IN_MONO ? 1 : 2);

int audio_bit_rate = 64000;
int audio_data_format = AudioFormat.ENCODING_PCM_16BIT;

m_audio_buffer_size = AudioRecord.getMinBufferSize(m_audio_sample_rate, m_audio_channels, audio_data_format) * 2;
m_audio_recorder = new AudioRecord(MediaRecorder.AudioSource.MIC, m_audio_sample_rate,
                                   m_audio_channels, audio_data_format, m_audio_buffer_size);

m_audio_encoder = MediaCodec.createEncoderByType(m_encode_audio_mime);
MediaFormat audio_format = new MediaFormat();
audio_format.setString(MediaFormat.KEY_MIME, m_encode_audio_mime);
audio_format.setInteger(MediaFormat.KEY_BIT_RATE, audio_bit_rate);
audio_format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, m_audio_channel_count);
audio_format.setInteger(MediaFormat.KEY_SAMPLE_RATE, m_audio_sample_rate);
audio_format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
audio_format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, m_audio_buffer_size);
m_audio_encoder.configure(audio_format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);

I found that the first time of AudioRecord.read() takes longer time to return, while the successive read() have time intervals that are more close to the real time of audio data. For example, my audio format is 44100Hz 16Bit 1Channel, and the buffer size of AudioRecord is 16384, so a full buffer means 185.76 ms. When I record the system time for each call of read() and subtracting them from a base time, I get the following sequence:

time before each read(): 0ms, 345ms, 543ms, 692ms, 891ms, 1093ms, 1244ms, ...

I feed these raw data to the audio encoder with the above time values as PTS, and the encoder outputs encoded audio samples with the following PTS:

encoder output PTS: 0ms, 185ms, 371ms, 557ms, 743ms, 928ms, ...

It looks like that the encoder treats each part of data as having the same time period. I believe that the encoder works correctly since I give it raw data with the same size (16384) every time. However, if I use the encoder output PTS as the input of muxer, I'll get a video with audio content being faster then video content.

I want to ask that:

1. Is it expected that the first time of AudioRecord.read() blocks longer? I'm sure that the function call takes more than 300ms while it only records 16384 bytes as 186ms. Is this also an issue that depends on device / Android version?
2. What should I do to achieve audio/video synchronization? I have a workaround to measure the delay time of the first call of read(), then shift the PTS of audio samples by the delay. Is there another better way to handle this?

解决方案

Convert the mono input to stereo. I was pulling my hair out for some time before I realised the AAC encoder exposed by MediaCoder only works with stereo input.

这篇关于如何正确处理PTS采用了Android AudioRecord和媒体codeC音频连接codeR？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！