Android MediaCodec在异步模式下比在同步模式下慢吗？ [英] Android MediaCodec slower in async-mode than in synchronous mode?

问题描述

关于我的关于orroids MediaCodec类的问题。我已成功设法解码原始h264内容并在两个TextureViews中显示结果。 h264流来自运行openGL场景的服务器。场景有一个摄像头，因此响应用户输入。为了进一步减少服务器上的输入和智能手机上的实际结果之间的延迟，我正在考虑使用MediaCodecs异步模式。
以下是我如何设置同步和异步两种变体：

again a question of mine regarding androids MediaCodec class. I have successfully managed to decode raw h264 content and display the result in two TextureViews. The h264 stream comes from a server that is running an openGL scene. The scene has a camera and is therefore responsive to users input. To further reduce the latency between an input on the server and the actual result on the smartphone I was thinking about using MediaCodecs Async mode. Here is how i set up both variants, synchronous and asynchronous:

异步：

//decoderCodec is "video/avc"
MediaFormat fmt = MediaFormat.createVideoFormat(decoderCodec, 1280,720);
codec.setCallback(new MediaCodec.Callback() {

    @Override
    public void onInputBufferAvailable(MediaCodec codec, int index) {
        byte[] frameData;
        try {
            frameData = frameQueue.take(); //this call is blocking
        } catch (InterruptedException e) {
            return;
        }

        ByteBuffer inputData = codec.getInputBuffer(index);
        inputData.clear();
        inputData.put(frameData);

        codec.queueInputBuffer(index, 0, frameData.length, 0, 0);
    }

    @Override
    public void onOutputBufferAvailable(MediaCodec codec, int index, MediaCodec.BufferInfo info) {
        codec.releaseOutputBuffer(index, true);
    }

     //The two other methods are left blank at the moment.

});


codec.configure(fmt, surface, null, 0);
codec.start();

同步：（设置为Async，但 codec.setCallback（...） part。两个变体所在的类都是Runnable。

Sync: (is setup like Async except the codec.setCallback(...) part. The class both variants reside in is Runnable.

public void run() {

    while(!Thread.interrupted())
    {
        if(!IS_ASYNC) {
            byte[] frameData;
            try {
                frameData = frameQueue.take(); //this call is blocking
            } catch (InterruptedException e) {
                break;
            }

            int inIndex = codec.dequeueInputBuffer(BUFFER_TIMEOUT);

            if (inIndex >= 0) {
                ByteBuffer input = codec.getInputBuffer(inIndex);
                input.clear();
                input.put(frameData);
                codec.queueInputBuffer(inIndex, 0, frameData.length, 0, 0);
            }

            MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
            int outIndex = codec.dequeueOutputBuffer(bufferInfo, BUFFER_TIMEOUT);

            if(outIndex >= 0)
                codec.releaseOutputBuffer(outIndex, true);
        }
        else sleep(3000); //Just for testing, if we are in Async, this thread has nothing to do actually...
    }
}

这两种方法都有效，但我观察到以同步模式播放的视频更加流畅，延迟也更低。

Both approaches work, but I'm observing that the videos played in synchronous-mode are much smoother and the latency is also lower.

我想出了使用异步模式的想法，因为 frameQueue 是一个 LinkedBlockingDeque 我想，如果同步解码器等待新帧数据到达的时间太长，则解码输出可能已经可用但由于队列的阻塞性质而未显示。另一方面，我不想做像忙等待和轮询队列，in​​putBuffers和outputBuffers一样的事情。

I came up with the idea of using the async mode because frameQueue is a LinkedBlockingDeque and I thought, if the synchronous decoder is waiting too long for new frame data to arrive, decoded output may already be available but is not displayed because of the blocking nature of the queue. On the other hand I dont wanted to do something like busy waiting and poll the queue, the inputBuffers and outputBuffers all the time.

所以我尝试使用Callbacks的AsyncMode，但是我得到的结果比同步模式更差。你们现在的问题是：为什么？我是否滥用异步模式或者其他什么？

So I tried the AsyncMode using the Callbacks, but the result I get is worse than in synchronous mode. The question to you guys now is: Why? Do I misuse the async mode or is it something else?

感谢您的反馈！

Christoph

Christoph

推荐答案

如果 onInputBufferAvailable 中的阻止调用是罪魁祸首，我不会感到惊讶。感觉很可能在同一个线程中调用了 onInputBufferAvailable 和 onOutputBufferAvailable ，如果你阻塞了一个，你就停止了另一个来自跑步。

I would not be surprised if the blocking call in onInputBufferAvailable is the culprit. It feels probable that both onInputBufferAvailable and onOutputBufferAvailable are called within the same thread, and if you block in one, you stop the other one from running.

我建议你改变它，以便你在 onInputBufferAvailable 中将缓冲区索引推到一些队列，并发出一个不同的线程，表示现在有另一个缓冲区可供使用，然后让第二个线程等待队列中的缓冲区，并在那里对输入数据进行阻塞获取。

I would suggest changing it so that you in onInputBufferAvailable just push the buffer index onto some queue, and signal a different thread that there's another buffer available now, then having this second thread wait for buffers from the queue, and do a blocking fetch of input data there.

这篇关于Android MediaCodec在异步模式下比在同步模式下慢吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！