MediaCodec同时编码和解码 [英] MediaCodec simultaneous encoding and decoding
问题描述
我正在尝试使用GPU将效果应用于视频的帧，然后将这些帧重新编码为新的结果视频.
I am trying to apply effects to the frames of a video using the GPU and then to re-encode those frames into a new result video.
为了提高性能，我执行了以下流程:
In the interest of performance I have implemented the following flow:
有3个不同的线程，每个线程都有自己的OpenGL上下文.这些上下文的设置方式使它们可以在它们之间共享纹理.
There are 3 different threads, each with it's own OpenGL context. These contexts are set up in such a way that they share textures between them.
线程1 从视频中提取帧并将其作为纹理保存在GPU内存中，类似于
Thread 1 extracts frames from the video and holds them in the GPU memory as textures, similar to this example.
线程2 使用修改后的GPUImage版本处理纹理，该版本还可以在GPU内存中输出纹理.
Thread 2 processes the textures using a modified version of GPUImage that also outputs textures in the GPU memory.
最后，线程3 将从线程2获得的纹理写入到新的视频文件中，类似于
Finally, thread 3 writes the textures obtained from thread 2 into a new video file similar to the method described here
使用线程1和2之间以及线程2和3之间的队列维护帧顺序. 之后，将纹理从内存中手动删除.
Frame order is maintained using queues between threads 1 and 2, and threads 2 and 3. Textures are deleted from memory manually after they are used for processing / writing.
此流程的重点是将每个进程分开，希望最终性能是3个线程中最慢的一个.
The whole point of this flow is to separate each process in hopes that the final performance will be that of the slowest of the 3 threads.
问题:
最终视频是90％的黑框，只有其中一些是正确的.
The final video is 90% black frames, only some of them being correct.
我检查了提取和处理的各个结果，它们均按预期工作.另外请注意，在3个线程中描述的3个组件在单个线程中可以很好地协同工作.
I have checked the individual results of extraction, and processing and they work as expected. Also note that the 3 components described in the 3 threads work just fine together in a single thread.
我尝试同步线程1和线程3，在为线程1添加额外的100ms睡眠时间后，视频结果还不错，可能有1或2个黑帧.在我看来，解码器和编码器这两个实例无法同时工作.
I have tried to synchronise thread 1 and thread 3, and after adding an extra 100ms sleep time to thread 1 the video turns out just fine, with maybe 1 or 2 black frames. Seems to me like the two instance of the decoder and encoder are unable to work simultaneously.
我将用所需的任何其他详细信息来编辑此帖子.
I will edit this post with any extra requested details.
推荐答案
在OpenGL ES上下文之间共享纹理需要格外小心. Grafika的显示+捕获相机"活动的实现方式已损坏；有关详细信息，请参见本期.基本的问题是，在更新纹理时，您基本上需要发出内存屏障.实际上，这意味着在生产者端发布glFinish()，并在消费者端重新绑定纹理，并在synchronized块中完成所有这些操作.
Sharing textures between OpenGL ES contexts requires some care. The way it's implemented in Grafika's "show + capture camera" Activity is broken; see this issue for details. The basic problem is that you essentially need to issue memory barriers when the texture is updated; in practical terms that means issuing glFinish() on the producer side and, and re-binding the texture on the consumer side, and doing all of this in synchronized blocks.
如果您可以在单个线程上完成所有的GLES工作，您的生活将变得更加简单(并且效率更高).以我的经验，一次激活多个GLES上下文是不明智的，并且通过找到替代方法可以避免一些麻烦.
Your life will be simpler (and more efficient) if you can do all of the GLES work on a single thread. In my experience, having more than one GLES context active at a time is unwise, and you'll save yourself some pain by finding an alternative.
您可能想要更多类似这样的东西:
You probably want something more like this:
- 线程#1读取文件并将帧馈送到MediaCodec解码器中.解码器将输出发送到SurfaceTexture Surface.
- 线程#2具有GLES上下文.它创建了线程1将输出发送到的SurfaceTexture.它处理图像并将输出呈现在MediaCodec编码器的Surface上.
- 创建MediaCodec编码器的线程#3坐在那里，等待编码输出.接收到输出后，将其写入磁盘.请注意，使用MediaMuxer可能会拖延；有关更多信息，请参见此博客文章.
在所有情况下，线程之间(以及内部进程)之间的唯一通信是通过Surface完成的. SurfaceTexture和MediaCodec实例是从单个线程创建和使用的.只有生产者端点(Surface)被传递.
In all cases, the only communication between threads (and, under the hood, processes) is done through Surface. The SurfaceTexture and MediaCodec instances are created and used from a single thread; only the producer endpoint (the Surface) is passed around.
一个潜在的麻烦点是流量控制-如果您太快地喂入它们，SurfaceTextures将掉落帧.结合线程#1和#2可能因情况而异.
One potential trouble point is with flow control -- SurfaceTextures will drop frames if you feed them too quickly. Combining threads #1 and #2 might make sense depending on circumstances.
这篇关于MediaCodec同时编码和解码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
