PCM ->AAC(编码器)->PCM(解码器)实时正确优化 [英] PCM -> AAC (Encoder) -> PCM(Decoder) in real-time with correct optimization
问题描述
我正在尝试实施
AudioRecord (MIC) ->
PCM -> AAC Encoder
AAC -> PCM Decode
-> AudioTrack?? (SPEAKER)
在 Android 4.1+ (API16) 上使用 MediaCodec.
with MediaCodec on Android 4.1+ (API16).
首先，我成功(但不确定是否正确优化)实现了PCM ->AAC 编码器 by MediaCodec 如下所示
Firstly, I successfully (but not sure correctly optimized) implemented PCM -> AAC Encoder by MediaCodec as intended as below
private boolean setEncoder(int rate)
{
encoder = MediaCodec.createEncoderByType("audio/mp4a-latm");
MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
format.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1024);//AAC-HE 64kbps
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectHE);
encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
return true;
}
输入:PCM 比特率 = 44100(Hz) x 16(bit) x 1(Monoral) = 705600 bit/s
输出:AAC-HE 比特率 = 64 x 1024(bit) = 65536 比特/秒
所以，数据大小大约压缩了 x11 ，我通过观察日志确认了这一点
So, the data size is approximately compressed x11 ,and I confirmed this working by observing a log
- AudioRecoder﹕读取了 4096 个字节
- AudioEncoder﹕ 369 字节编码
数据大小大约压缩了 x11，到目前为止一切都很好.
the data size is approximately compressed x11, so far so good.
现在，我有一个 UDP 服务器来接收编码数据，然后对其进行解码.
Now, I have a UDP server to receive the encoded data, then decode it.
解码器配置文件设置如下:
The decoder profile is set as follows:
private boolean setDecoder(int rate)
{
decoder = MediaCodec.createDecoderByType("audio/mp4a-latm");
MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
format.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1024);//AAC-HE 64kbps
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectHE);
decoder.configure(format, null, null, 0);
return true;
}
由于 UDPserver 数据包缓冲区大小为 1024
Since UDPserver packet buffer size is 1024
- UDPserver ﹕ 收到 1024 个字节
由于这是压缩的 AAC 数据，我希望解码大小为
and since this is the compressed AAC data, I would expect the decoding size will be
大约 1024 x11，但实际结果是
approximately 1024 x11, however the actual result is
- AudioDecoder: 8192 字节解码
大约是 x8，我觉得有些不对劲.
It's approximately x8, and I feel something wrong.
解码器代码如下:
IOudpPlayer = new Thread(new Runnable()
{
public void run()
{
SocketAddress sockAddress;
String address;
int len = 1024;
byte[] buffer2 = new byte[len];
DatagramPacket packet;
byte[] data;
ByteBuffer[] inputBuffers;
ByteBuffer[] outputBuffers;
ByteBuffer inputBuffer;
ByteBuffer outputBuffer;
MediaCodec.BufferInfo bufferInfo;
int inputBufferIndex;
int outputBufferIndex;
byte[] outData;
try
{
decoder.start();
isPlaying = true;
while (isPlaying)
{
try
{
packet = new DatagramPacket(buffer2, len);
ds.receive(packet);
sockAddress = packet.getSocketAddress();
address = sockAddress.toString();
Log.d("UDP Receiver"," received !!! from " + address);
data = new byte[packet.getLength()];
System.arraycopy(packet.getData(), packet.getOffset(), data, 0, packet.getLength());
Log.d("UDP Receiver", data.length + " bytes received");
//===========
inputBuffers = decoder.getInputBuffers();
outputBuffers = decoder.getOutputBuffers();
inputBufferIndex = decoder.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0)
{
inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(data);
decoder.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0);
}
bufferInfo = new MediaCodec.BufferInfo();
outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0);
while (outputBufferIndex >= 0)
{
outputBuffer = outputBuffers[outputBufferIndex];
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
Log.d("AudioDecoder", outData.length + " bytes decoded");
decoder.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0);
}
//===========
}
catch (IOException e)
{
}
}
decoder.stop();
}
catch (Exception e)
{
}
}
});
完整代码:
https://gist.github.com/kenokabe/9029256
还需要权限:
<uses-permission android:name="android.permission.INTERNET"></uses-permission>
<uses-permission android:name="android.permission.RECORD_AUDIO"></uses-permission>
<小时>
一位在 Google 工作的成员 fadden 告诉我
看起来我没有设置位置 &限制输出缓冲区.
Looks like I'm not setting position & limit on the output buffer.
我读过VP8 编码 Nexus 5 返回空/0 帧 ，但不确定如何正确实现.
I have read VP8 Encoding Nexus 5 returns empty/0-Frames , but not sure how to implement correctly.
更新:我有点明白在哪里修改
看起来我没有设置位置 &限制输出缓冲区.
Looks like I'm not setting position & limit on the output buffer.
，所以在Encoder和Decoder的while循环中添加2行如下:
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
https://gist.github.com/kenokabe/9029256/revisions
结果是一样的.
现在，我认为，错误:W/SoftAAC2:AAC 解码器返回错误 16388，替代静音. 表示此解码器从第一个开始就完全失败了.这又是数据不可搜索问题.在 Android 上寻找 AAC 流 如果 AAC 解码器不能处理以这种方式流式传输数据，但只需要添加一些标题.
and now, I think, the errors:
W/SoftAAC2﹕ AAC decoder returned error 16388, substituting silence. indicates this decoder fails completely from the first. It's again the data is not seekable issue. Seeking in AAC streams on Android It's very disappointing if the AAC decoder cannot handle the streaming data in this way but only with adding some header.
UPDATE2:UDP 接收器做错了，所以修改
https://gist.github.com/kenokabe/9029256
现在，错误
W/SoftAAC2:AAC解码器返回错误16388，替代静音.消失了！！
W/SoftAAC2﹕ AAC decoder returned error 16388, substituting silence.
disappeared!!
所以，这表明解码器至少可以正常工作，
So, it indicates the decoder works without an error, at least,
然而，这是1个周期的日志:
however, this is the log of 1 cycle:
D/AudioRecoder﹕ 4096 bytes read
D/AudioEncoder﹕ 360 bytes encoded
D/UDP Receiver﹕ received !!! from /127.0.0.1:39000
D/UDP Receiver﹕ 360 bytes received
D/AudioDecoder﹕ 8192 bytes decoded
PCM(4096)->AACencoded(360)->UDP-AAC(360)->(应该是)PCM(8192)
PCM(4096)->AACencoded(360)->UDP-AAC(360)->(supposed to be )PCM(8192)
最终结果大约是原始 PCM 大小的 2 倍，还是有问题.
The final result is about 2x size of the original PCM, something is still wrong.
所以我的问题是
你能正确优化我的示例代码以使其正常工作吗?
Can you properly optimize my sample code to work correctly?
使用 AudioTrack API 即时播放解码的 PCM 原始数据是否正确?您能告诉我正确的方法吗?示例代码表示赞赏.
Is it a right way to use AudioTrack API to play the decoded PCM raw data on the fly, and can you show me the proper way to do that? A example code is appreciated.
谢谢.
附注.我的项目目标是 Android4.1+(API16)，我读过 API18(Andeoid 4.3+) 上的东西更容易，但出于明显的兼容性原因，不幸的是，我不得不在这里跳过 MediaMuxer 等...
PS. My project targets on Android4.1+(API16), I've read things are easier on API18(Andeoid 4.3+), but for obvious compatibility reasons, unfortunately, I have to skip MediaMuxer etc. here...
推荐答案
经过测试，这是我通过修改您的代码得出的结论:
After testing this is what I came up with from modifying your code:
package com.example.app;
import android.app.Activity;
import android.media.AudioManager;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.os.Bundle;
import android.media.AudioFormat;
import android.media.AudioRecord;
import android.media.AudioTrack;
import android.media.MediaCodec;
import android.media.MediaRecorder.AudioSource;
import android.util.Log;
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.SocketAddress;
import java.net.SocketException;
import java.nio.ByteBuffer;
public class MainActivity extends Activity
{
private AudioRecord recorder;
private AudioTrack player;
private MediaCodec encoder;
private MediaCodec decoder;
private short audioFormat = AudioFormat.ENCODING_PCM_16BIT;
private short channelConfig = AudioFormat.CHANNEL_IN_MONO;
private int bufferSize;
private boolean isRecording;
private boolean isPlaying;
private Thread IOrecorder;
private Thread IOudpPlayer;
private DatagramSocket ds;
private final int localPort = 39000;
@Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
IOrecorder = new Thread(new Runnable()
{
public void run()
{
int read;
byte[] buffer1 = new byte[bufferSize];
ByteBuffer[] inputBuffers;
ByteBuffer[] outputBuffers;
ByteBuffer inputBuffer;
ByteBuffer outputBuffer;
MediaCodec.BufferInfo bufferInfo;
int inputBufferIndex;
int outputBufferIndex;
byte[] outData;
DatagramPacket packet;
try
{
encoder.start();
recorder.startRecording();
isRecording = true;
while (isRecording)
{
read = recorder.read(buffer1, 0, bufferSize);
// Log.d("AudioRecoder", read + " bytes read");
//------------------------
inputBuffers = encoder.getInputBuffers();
outputBuffers = encoder.getOutputBuffers();
inputBufferIndex = encoder.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0)
{
inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(buffer1);
encoder.queueInputBuffer(inputBufferIndex, 0, buffer1.length, 0, 0);
}
bufferInfo = new MediaCodec.BufferInfo();
outputBufferIndex = encoder.dequeueOutputBuffer(bufferInfo, 0);
while (outputBufferIndex >= 0)
{
outputBuffer = outputBuffers[outputBufferIndex];
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
// Log.d("AudioEncoder ", outData.length + " bytes encoded");
//-------------
packet = new DatagramPacket(outData, outData.length,
InetAddress.getByName("127.0.0.1"), localPort);
ds.send(packet);
//------------
encoder.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = encoder.dequeueOutputBuffer(bufferInfo, 0);
}
// ----------------------;
}
encoder.stop();
recorder.stop();
}
catch (Exception e)
{
e.printStackTrace();
}
}
});
IOudpPlayer = new Thread(new Runnable()
{
public void run()
{
SocketAddress sockAddress;
String address;
int len = 2048
byte[] buffer2 = new byte[len];
DatagramPacket packet;
byte[] data;
ByteBuffer[] inputBuffers;
ByteBuffer[] outputBuffers;
ByteBuffer inputBuffer;
ByteBuffer outputBuffer;
MediaCodec.BufferInfo bufferInfo;
int inputBufferIndex;
int outputBufferIndex;
byte[] outData;
try
{
player.play();
decoder.start();
isPlaying = true;
while (isPlaying)
{
try
{
packet = new DatagramPacket(buffer2, len);
ds.receive(packet);
sockAddress = packet.getSocketAddress();
address = sockAddress.toString();
// Log.d("UDP Receiver"," received !!! from " + address);
data = new byte[packet.getLength()];
System.arraycopy(packet.getData(), packet.getOffset(), data, 0, packet.getLength());
// Log.d("UDP Receiver", data.length + " bytes received");
//===========
inputBuffers = decoder.getInputBuffers();
outputBuffers = decoder.getOutputBuffers();
inputBufferIndex = decoder.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0)
{
inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(data);
decoder.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0);
}
bufferInfo = new MediaCodec.BufferInfo();
outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0);
while (outputBufferIndex >= 0)
{
outputBuffer = outputBuffers[outputBufferIndex];
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
// Log.d("AudioDecoder", outData.length + " bytes decoded");
player.write(outData, 0, outData.length);
decoder.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 0..
这篇关于PCM ->AAC(编码器)->PCM(解码器)实时正确优化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
