Android MediaCodec 实时 h264 编码/解码延迟 [英] Android MediaCodec realtime h264 encoding/decoding latency

问题描述

我正在使用 Android MediaCodec 并将其用于来自相机的实时 H264 编码和解码帧.我以同步方式使用 MediaCodec 并将输出渲染到解码器的 Surface 并且一切正常，除了我的实时延迟很长，需要 1.5-2 秒，我很困惑为什么会这样.我测量了编码和解码过程的总时间，它保持在 50-65 毫秒左右，所以我认为问题不在于它们.我尝试更改编码器的配置，但没有帮助，目前配置如下:

I'm working with Android MediaCodec and use it for a realtime H264 encoding and decoding frames from camera. I use MediaCodec in synchronous manner and render the output to the Surface of decoder and everething works fine except that I have a long latency from a realtime, it takes 1.5-2 seconds and I'm very confused why is it so. I measured a total time of encoding and decoding processes and it keeps around 50-65 milliseconds so I think the problem isn't in them. I tried to change the configuration of the encoder but it didn't help and currently it configured like this:

val formatEncoder = MediaFormat.createVideoFormat("video/avc", 1920, 1080)
formatEncoder.setInteger(MediaFormat.KEY_FRAME_RATE, 30)
formatEncoder.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5)
formatEncoder.setInteger(MediaFormat.KEY_BIT_RATE, 1920 * 1080)
formatEncoder.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface)
val encoder = MediaCodec.createEncoderByType("video/avc")
encoder.configure(formatEncoder, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
val inputSurface = encoder.createInputSurface() // I use it to send frames from camera to encoder
encoder.start()

更改解码器的配置也对我没有任何帮助，目前我是这样配置的:

Changing the configuration of the decoder also didn't help me at all and currently I configured it like this:

val formatDecoder = MediaFormat.createVideoFormat("video/avc", 1920, 1080)
val decoder = MediaCodec.createDecoderByType("video/avc")
decoder.configure(formatDecoder , outputSurface, null, 0) // I use outputSurface to render decoded frames into it
decoder.start()

我使用以下超时来等待可用的编码器/解码器缓冲区，我试图减少它们的值，但它没有帮助我，我这样离开它们:

I use the following timeouts for waiting for available encoder/decoder buffers I tried to reduce their values but it didn't help me and I left them like this:

var TIMEOUT_IN_BUFFER = 10000L // microseconds
var TIMEOUT_OUT_BUFFER = 10000L // microseconds

我还测量了消耗 inputSurface 一帧的时间，这次需要 0.03-0.05 毫秒，所以它不是瓶颈.实际上，我测量了所有可能存在瓶颈的地方，但没有发现任何问题，我认为问题出在编码器或解码器本身或其配置中，或者我应该使用一些特殊的例程将帧发送到编码/解码..

Also I measured the time of consuming the inputSurface a frame and this time takes 0.03-0.05 milliseconds so it isn't a bottleneck. Actually I measured all the places where a bottleneck could be, but I wasn't found anything and I think the problem is in the encoder or decoder itself or in their configurations, or maybe I should use some special routine for sending frames to encoding/decoding..

我还尝试使用硬件加速编解码器，这是唯一对我有帮助的方法，当我使用它时，延迟减少到约 500-800 毫秒，但它仍然不适合我进行实时流式传输.

I also tried to use HW accelerated codec and it's the only thing that helped me, when I use it the latency reduces to ~ 500-800 milliseconds but it still doesn't fit me for a realtime streaming.

在我看来，编码器或解码器在开始在表面上显示之前会缓冲几帧，最终会导致延迟，如果确实如此，我该如何禁用缓冲或减少缓冲时间?

It seems to me that the encoder or decoder buffers several frames before start displaying them on the surface and eventually it leads to the latency and if it really so then how can I disable bufferization or reduce the time of it?

请帮帮我，我在这个问题上坚持了大约半年，不知道如何减少延迟，我确信这是可能的，因为 Telegram、Viber、WhatsApp 等流行应用程序运行良好，没有延迟那么这里的秘密是什么?

Please help me I'm stucking on this problem for about half a year and have no idea how to reduce the latency, I'm sure that it's possible because popular apps like Telegram, Viber, WhatsApp etc. work fine and without latency so what's the secret here?

UPD 07.07.2021:

我仍然没有找到摆脱延迟的解决方案.我尝试更改 h264 配置文件，增加和减少 I 帧间隔、比特率、帧率，但结果相同，唯一有助于减少延迟的方法 - 将分辨率从 1920x1080 降级到例如640x480，但是这个解决方案"不适合我，因为我想编码/解码分辨率为 1920x1080 的实时视频.

I still haven't found a solution to get rid of the latency. I've tried to change h264 profiles, increase and decrease I-frame inteval, bitrate, framerate, but result the same, the only thing that hepls a little to reduce the latency - downgrade the resolution from 1920x1080 to e.g. 640x480, but this "solution" doesn't suit me because I want to encode/decode a realtime video with 1920x1080 resolution.

UPD 08.07.2021:

我发现，如果我将 TIMEOUT_IN_BUFFER 和 TIMEOUT_OUT_BUFFER 的值从 10_000L 更改为 100_000L，它会稍微减少延迟，但会在开始编码/解码过程后显着增加显示第一帧的延迟.

I found out that if I change the values of TIMEOUT_IN_BUFFER and TIMEOUT_OUT_BUFFER from 10_000L to 100_000L it decreases the latency a bit but increases the delay of showing the first frame quite a lot after start encoding/decoding process.

推荐答案

我相信 android h264 解码器有延迟(至少在大多数情况下我已经尝试过).可能这就是 Android 开发人员从 API 级别 30 添加 PARAMETER_KEY_LOW_LATENCY 的原因.但是，我可以通过多次查询输出来减少某些帧的延迟.理由:不知道.这只是无聊的试验和错误的结果

I believe android h264 decoder have latency (at-least in most cases i've tried). Probably that's why android developers added PARAMETER_KEY_LOW_LATENCY from API level 30. However I could decrease the delay some frames by querying for the output some more times. Reason: no idea. It's just result of boring trial and errors

int inputIndex = m_codec.dequeueInputBuffer(-1);// Pass in -1 here bc we don't have a playback time reference

if (inputIndex >= 0) {
    ByteBuffer buffer;
    if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.LOLLIPOP) {
        buffer = m_codec.getInputBuffer(inputIndex);
    } else {
        ByteBuffer[] bbuf = m_codec.getInputBuffers();
        buffer = bbuf[inputIndex];
    }
    buffer.put(frame);

    // tell the decoder to process the frame
    m_codec.queueInputBuffer(inputIndex, 0, frame.length, 0, 0);
}
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();

int outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
    m_codec.releaseOutputBuffer(outputIndex, true);
}
outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
    m_codec.releaseOutputBuffer(outputIndex, true);
}
outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
    m_codec.releaseOutputBuffer(outputIndex, true);
}

这篇关于Android MediaCodec 实时 h264 编码/解码延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！