控制 VirtualDisplay 的帧率 [英] Controlling Frame Rate of VirtualDisplay

问题描述

我正在编写一个 Android 应用程序，其中有一个 VirtualDisplay 来镜像屏幕上的内容，然后我将帧从屏幕发送到 的实例媒体编解码器.它可以工作，但是，我想添加一种指定编码视频的 FPS 的方法，但我不确定该怎么做.

I'm writing an Android application, and in it, I have a VirtualDisplay to mirror what is on the screen and I then send the frames from the screen to an instance of a MediaCodec. It works, but, I want to add a way of specifying the FPS of the encoded video, but I'm unsure how to do so.

根据我的阅读和试​​验，丢弃编码帧(基于演示时间)效果不佳，因为它最终会产生块状/伪影视频，而不是低帧率的流畅视频.其他阅读表明，做我想做的事(限制 FPS)的唯一方法是将传入的 FPS 限制为 MediaCodec，但 VirtualDisplay 只接收一个 Surface 由 MediaCodec 构造而成，如下

From what I've read and experimented with, dropping encoded frames (based on the presentation times) doesn't work well as it ends up with blocky/artifact ridden video as opposed to a smooth video at a lower framerate. Other reading suggests that the only way to do what I want (limit the FPS) would be to limit the incoming FPS to the MediaCodec, but the VirtualDisplay just receives a Surface which is constructed from the MediaCodec as below

mSurface = <instance of MediaCodec>.createInputSurface();
mVirtualDisplay = mMediaProjection.createVirtualDisplay(
    "MyDisplay",
    screenWidth,
    screenHeight,
    screenDensity,
    DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
    mSurface,
    null,
    null);

我还尝试将 Surface 子类化并限制通过 unlockCanvasAndPost(Canvas canvas) 提供给 MediaCodec 的帧，但是函数似乎从未在我的实例上被调用，因此，我如何扩展 Surface 以及与 Parcel 作为 writeToParcel 的交互可能有些奇怪> 函数被在我的实例上调用，但这是我的实例中唯一调用的函数(我可以说).

I've also tried subclassing Surface and limit the frames that are fed to the MediaCodec via the unlockCanvasAndPost(Canvas canvas) but the function never seems to be called on my instance, so, there may be some weirdness in how I extended Surface and the interaction with the Parcel as writeToParcel function is called on my instance, but that is the only function that is called in my instance (that I can tell).

其他阅读建议我可以从编码器 -> 解码器 -> 编码器开始，并限制第二个编码器接收帧的速率，但如果可以避免的话，我宁愿不做很多额外的计算.

Other reading suggests that I can go from encoder -> decoder -> encoder and limit the rate in which the second encoder is fed frames, but that's a lot of extra computation that I'd rather not do if I can avoid it.

是否有人成功地限制了 VirtualDisplay 提供其 Surface 的速率?任何帮助将不胜感激！

Has anyone successfully limited the rate at which a VirtualDisplay feeds its Surface? Any help would be greatly appreciated!

推荐答案

从你不能做的开始...

您不能从编码流中删除内容.编码流中的大多数帧本质上与其他帧不同".如果不知道帧是如何交互的，就无法安全地删除内容，最终会出现损坏的宏块外观.

You can't drop content from the encoded stream. Most of the frames in the encoded stream are essentially "diffs" from other frames. Without knowing how the frames interact, you can't safely drop content, and will end up with that corrupted macroblock look.

您无法为 MediaCodec 编码器指定帧速率.它可能会将其填充到某个地方的元数据中，但对编解码器而言唯一真正重要的是您输入的帧，以及与每个帧相关的演示时间戳.编码器不会丢帧.

You can't specify the frame rate to the MediaCodec encoder. It might stuff that into metadata somewhere, but the only thing that really matters to the codec is the frames you're feeding into it, and the presentation time stamps associated with each frame. The encoder will not drop frames.

通过继承 Surface 无法做任何有用的事情.Canvas 操作仅用于软件渲染，与从相机或虚拟显示器输入帧无关.

You can't do anything useful by subclassing Surface. The Canvas operations are only used for software rendering, which is unrelated to feeding in frames from a camera or virtual display.

您可以做的是将帧发送到中间 Surface，然后选择是否将它们转发到 MediaCodec 的输入 Surface.一种方法是创建一个 SurfaceTexture，从中构造一个 Surface，然后将其传递给虚拟显示器.当 SurfaceTexture 的帧可用回调触发时，您要么忽略它，要么使用 GLES 将纹理渲染到 MediaCodec 输入 Surface 上.

What you can do is send the frames to an intermediate Surface, and then choose whether or not to forward them to the MediaCodec's input Surface. One approach would be to create a SurfaceTexture, construct a Surface from it, and pass that to the virtual display. When the SurfaceTexture's frame-available callback fires, you either ignore it, or render the texture onto the MediaCodec input Surface with GLES.

各种示例可以在 Grafika 和 bigflake，没有一个是完全合适的，但所有必要的 EGL 和 GLES 类都在那里.

Various examples can be found in Grafika and on bigflake, none of which are an exact fit, but all of the necessary EGL and GLES classes are there.

这篇关于控制 VirtualDisplay 的帧率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！