缓冲表面输入到 MediaCodec [英] Buffering Surface input to MediaCodec

问题描述

已经演示了如何像 CameraPreview 一样为 MediaCodec 提供 Surface 输入，但是在提交到 MediaCodec 之前，有没有实用的方法来缓冲这个输入?

It's been demonstrated how to feed MediaCodec with Surface input like the CameraPreview, but are there practical ways of buffering this input before submission to MediaCodec?

在我的实验中，Galaxy Nexus 在制作音频/视频时遇到了令人无法接受的问题使用 CameraToMpegTest.java

In my experiments, a Galaxy Nexus experiences unacceptable hiccups in producing audio / video streams using the direct, synchronous encoding method in CameraToMpegTest.java

当使用 MediaCodec 和 byte[] 或 ByteBuffer 输入时，我们可以将未编码的数据提交给 ExecutorService或类似的处理队列，以确保不丢帧，即使设备遇到我们应用程序无法控制的 CPU 使用率峰值.但是，由于需要执行Android 的相机和 MediaCodec 之间的颜色格式转换这种方法对于高分辨率的实时视频是不现实的.

When using MediaCodec with byte[] or ByteBuffer input, we can submit unencoded data to a ExecutorService or similar queue for processing to ensure no frames are dropped, even if the device experiences spikes in CPU usage out of our application's control. However, due to the requirement of performing color format conversion between Android's Camera and MediaCodec this method is unrealistic for high resolution, live video.

想法:

1. 有没有办法提供用 EGL14.eglCopyBuffers(EGLDisplay d, EGLSurface s, NativePixmapType p) 到 MediaCodec?

1. Is there a way to feed the NativePixmapType created with EGL14.eglCopyBuffers(EGLDisplay d, EGLSurface s, NativePixmapType p) to MediaCodec?

Android 中的任何人都可以评论协调相机和 MediaCodec 之间的 ByteBuffer 格式是否在路线图上?

Can anyone from Android comment on whether harmonizing ByteBuffer formats between the Camera and MediaCodec is on the roadmap?

推荐答案

您根本不想复制数据.为大量数据分配存储和复制可能需要足够长的时间来降低帧速率.这通常排除了 byte[] 和 ByteBuffer[] 解决方案，即使您不必进行 U/V 平面交换.

You really don't want to copy the data at all. Allocating storage for and copying a large chunk of data can take long enough to kill your frame rate. This generally rules out byte[] and ByteBuffer[] solutions, even if you didn't have to do a U/V plane swap.

在系统中移动数据的最有效方法是使用 Surface.诀窍是 Surface 不是缓冲区，它是 缓冲区队列.缓冲区通过引用传递；当您unlockCanvasAndPost() 时，您实际上是将当前缓冲区放入消费者队列中，而消费者通常处于不同的进程中.

The most efficient way to move data through the system is with a Surface. The trick is that a Surface isn't a buffer, it's an interface to a queue of buffers. The buffers are passed around by reference; when you unlockCanvasAndPost() you're actually placing the current buffer onto a queue for the consumer, which is often in a different process.

没有公共机制来创建新的缓冲区并将其添加到队列使用的集合中，或者从队列中提取缓冲区，因此您无法在侧面实现DIY缓冲方案.没有用于更改池中缓冲区数量的公共接口.

There is no public mechanism for creating a new buffer and adding it to the set used by the queue, or for extracting buffers from the queue, so you can't implement a DIY buffering scheme on the side. There's no public interface to change the number of buffers in the pool.

了解导致打嗝的原因会很有用.用于分析此类问题的 Android 工具是 systrace，可在 Android 4.1+ 中使用(docs, 示例, bigflake 示例).如果您可以确定 CPU 负载的来源，或者确定它不是 CPU 而是一些代码混乱，那么您可能会有一个比向 Surface 添加更多缓冲区更容易的解决方案.

It'd be useful to know what it is that's causing the hiccups. The Android tool for analyzing such issues is systrace, available in Android 4.1+ (docs, example, bigflake example). If you can identify the source of the CPU load, or determine that it's not CPU but rather some bit of code getting tangled up, you'll likely have a solution that's much easier than adding more buffers to Surface.

这篇关于缓冲表面输入到 MediaCodec的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！