最小化的Android GLSurfaceView滞后 [英] Minimize Android GLSurfaceView lag
问题描述
继堆栈溢出一些其他的问题，我读过的指南Android的表面，SurfaceViews等从这里内部：
<一个href="https://source.android.com/devices/graphics/architecture.html">https://source.android.com/devices/graphics/architecture.html
这是导给了我所有的不同组件如何相互配合Android上一个大大改善的理解。它涵盖了如何eglSwapBuffers只是推渲染帧到稍后将被SurfaceFlinger被消耗时，prepares下一帧的显示队列。如果队列是满的，则它将等待直到一缓冲器返回之前为下一帧变为可用。上述文献描述此为馅队列，依靠交换缓冲器的背pressure到再现限制到显示器的垂直同步。这是使用默认的连续呈现GLSurfaceView模式会发生什么。
如果您的呈现是简单并完成在比帧周期要少得多，这样做的负面影响是引起BufferQueue一个附加的滞后，如在SwapBuffers等待不会发生，直到队列是满的，并且因此我们正在渲染帧总是注定要成为在队列的后面，并因此将不会马上在下一垂直同步，因为有前可能缓冲器队列中显示的
在此相反渲染点播通常发生的频率大大低于显示器的更新速度，因此通常BufferQueues的这些观点是空的，因此任何更新推入这些队列会被SurfaceFlinger在第二天VSYNC被抓住了。
因此，这里的问题：我如何建立一个持续的渲染，但最小的滞后？的目标是，在缓冲队列是空的，在每个垂直同步开始时，我使我在根据16ms的内容，将其推到队列（缓冲器计数= 1），且随后消耗SurfaceFlinger上的下一个垂直同步（缓冲器计数= 0），重复。缓冲器中的队列的数目可以看出，在systrace，所以目标是有0和1之间此替代
我上面提到的文件介绍编舞，以此来获得每个垂直同步回调。但是我不认为这是足以能实现最小的滞后行为，我以后。我已经测试做一个requestRender（）在一个垂直同步回调一个非常小的onDrawFrame（），它确实表现出0/1缓冲器计数行为。但是，如果SurfaceFlinger不能将一帧期间内完成所有的工作（可能是通知或其他持久性有机污染物）？在这种情况下，我希望我的渲染会很乐意将生产每VSYNC 1架，但BufferQueue的消费终端已经下降了框架。结果：我们现在交替缓冲器1和2之间在我们的队列中，我们已经取得了这样的渲染和看到的框架之间滞后的框架
。该文件似乎表明在看的时候的报道VSYNC时间和回调时运行之间的偏移。我可以看到，它可以帮助，如果你的回调延期交付是由于你的主线程由于布局传递什么的。不过，我不认为这将允许检测SurfaceFlinger的慢半拍，未能消耗的框架。有没有什么办法的应用程序可以制定出SurfaceFlinger已经下降了框架？它也似乎不能告诉队列的长度打破了使用垂直同步时间的游戏状态更新的想法，因为有一个在一个你正在渲染将实际显示之前的队列数目不详的帧。
减小队列的最大长度，并依托后台pressure将是实现这一目标的一种方式，但我不认为有一个API来设置缓冲区的GLSurfaceView BufferQueue的最大数量<？ / P>
大问题。
背景其他任何人阅读本快速位：
这里的目标是最小化显示延迟，即当应用程序呈现的帧之间的时间时，在显示面板点亮的像素。如果你只是扔内容在屏幕上，也不要紧，因为用户无法分辨。如果你正在响应触摸输入，但是，延迟的每一帧，使您的应用程序感觉只是有点不太适应。
现在的问题是相似的A / V同步，你需要有一个帧相关出来说话的视频帧音频被显示在屏幕上。在这种情况下，整体延迟并不重要，只要其始终对音频和视频输出相等。这面临着非常相似的问题，虽然，因为如果SurfaceFlinger摊位和您的视频始终显示一帧后，你将失去同步。
SurfaceFlinger运行在升高的优先级，并执行相对小的工作，所以不太可能错过任何一个节拍自身...但它可能发生。此外，它是合成来自多个源，其中的一些使用围栏信号异步完成帧。如果对实时视频帧由用OpenGL输出，GLES呈现尚未完成时的最后期限命中，整个构图则顺延至下一个VSYNC。
，以减少等待时间的愿望是足够强大的Android的奇巧（4.4）版本中引入的SurfaceFlinger的DispSync功能，剃延迟的半帧离开通常的两帧延迟。 （这是在图形架构文档简要提及，但它不是在wides $ P $垫使用。）
这就是这种情况。在过去，这是少一个问题的视频，因为30fps的视频更新每隔一个帧。打嗝工作本身出自然的，因为我们并没有试图保持队列满。我们开始看到为48Hz及60Hz的视频了，所以这更重要。
现在的问题是，我们如何检测，如果显示是送我们到SurfaceFlinger帧尽快，或者花费额外的帧等待缓冲的背后，我们送previously？
答案的第一部分是：你不能。没有状态的查询或回调的SurfaceFlinger，会告诉你什么样的状态。从理论上讲，你可以查询BufferQueue本身，但不一定会告诉你什么是你需要知道的。
与查询和回调的问题是，他们不能告诉你的状态的是的，只有什么状态的已的。由该应用接收该信息并作用在其上的时间，这种情况可能是完全不同的。该应用程序将在普通优先级运行，所以它受到延误。
有关A / V同步它稍微复杂一些，因为应用程序无法知道的显示特性。例如，一些显示器具有智能板有内存内置于他们。 （如果有什么在屏幕上不经常更新，可以节省大量的电能通过不具有面板扫描整个每秒内存总线60X像素）。这些会增加延迟，必须占一个额外的框架。
Android正在走向对A / V同步的解决方案是让应用程序告诉SurfaceFlinger当它要显示的帧。如果SurfaceFlinger错过了最后期限，则丢弃该帧。这是加入实验在4.4，虽然它不是真的打算使用，直到下一个版本（它应该工作不够好L preVIEW，虽然我不知道这是否包括所有使用所需的片它充分）。
该方法的应用程序使用，这是调用 EGL presentationTimeANDROID（）在 eglSwapBuffers扩展（）。该参数的功能所需的presentation时间，以纳秒为单位，使用相同的时基作为编舞（特别是Linux的 CLOCK_MONOTONIC ）。因此，对于每一帧，你把你的编舞得到了时间戳，加帧乘以近似的刷新率（所需数量，您可以通过查询显示对象获取 - 看<一href="https://github.com/google/grafika/blob/master/src/com/android/grafika/MiscUtils.java">MiscUtils#getDisplayRefreshNsec()),并把它传递到EGL，当你交换缓冲，所需的presentation时间随缓冲过去了。
回想SurfaceFlinger一次唤醒每VSYNC，着眼于未决缓冲器的收集，并通过硬件作曲提供了一组到显示器的硬件。如果你要求显示在时间t，SurfaceFlinger认为，传递到显示硬件的框架将显示在时间T-1或更早版本，该框架将举行（和previous帧重新显示）。如果帧将出现在时间T，它将被发送到显示器。如果帧将出现在时间T + 1或更高版本（即，它会错过时间期限），和的还有它背后定于稍后时间排队的另一帧（如帧供时间T + 1），则该帧用于时间T将被丢弃。
该解决方案并不完全适合您的问题。对于A / V同步，你需要不断的延迟，并不是最小的延迟。如果你看一下Grafika的<一个href="https://github.com/google/grafika/blob/master/src/com/android/grafika/ScheduledSwapActivity.java">scheduled交换活动中，你可以发现，使用 EGL presentationTimeANDROID（）在类似于视频播放器会做这样一些code。 （在当前状态下，它比音调生成器，用于创建systrace输出更小，但基本部分都在那里。）战略有呈现几帧前进，所以SurfaceFlinger不会枯竭，但是这恰恰错了你应用程序。
在presentation时机制，确实，但是，提供一种方式来丢弃帧，而不是让他们回来了。如果你碰巧知道，有报道的编导以及何时可以显示你的帧的时间之间的时间延迟两帧，您可以使用此功能，以确保帧将被丢弃，而不是排队，如果他们太远了过去。该Grafika活动，您可以设置帧速率和时延要求，然后查看systrace结果。
这将是有益的一个应用程序知道有多少延迟SurfaceFlinger帧实际上有，但没有一个查询的。 （这是有点尴尬，处理，无论如何，智能板可以改变模式，从而改变显示的等待时间;但除非你工作的A / V同步，你真正关心的是最小化SurfaceFlinger延迟。）这是合理的安全承担4.3+上的两帧。如果不是两帧，你可能有不理想的表现，但实际效果会不会比你会得到，如果你没有设置presentation的时间都没有。
您可以尝试设置所需的presentation时间等于编排时间戳;在最近的过去的时间戳的意思是展示的ASAP。这可确保最小的延迟，但是可以平滑适得其反。 SurfaceFlinger有两帧的延迟，因为它使一切都在系统中有足够的时间来完成工作。如果您的工作负载是不平衡的，你会单帧和双帧延时之间摆动，且输出看起来janky的转变。 （这是用于DispSync，这减少了总的时间至1.5帧一个关注的问题。）
我不记得 EGL presentationTimeANDROID（）函数添加的时候，但在老版本应该是一个空操作。
底线：为L，并在一定程度上4.4，你应该能够得到你想要使用EGL扩展与延迟两帧的行为。在早期版本中没有从系统的帮助。如果你想确保有没有在您的方式缓冲，你可以故意丢弃的帧，每隔一段时间，让缓冲队列排。
Following some other questions on Stack Overflow, I've read the guide to the internals of Android Surfaces, SurfaceViews, etc from here:
https://source.android.com/devices/graphics/architecture.html
That guide has given me a much improved understanding of how all the different pieces fit together on Android. It covers how eglSwapBuffers just pushes the rendered frame into a queue which will later be consumed by SurfaceFlinger when it prepares the next frame for display. If the queue is full, then it will wait until a buffer becomes available for the next frame before returning. The document above describes this as "stuffing the queue" and relying on the "back-pressure" of swap buffers to limit the rendering to the vsync of the display. This is what happens using the default continuous render mode of the GLSurfaceView.
If your rendering is simple and completes in much less than the frame period, the negative effect of this is an additional lag caused by the BufferQueue, as the wait on SwapBuffers doesn't happen until the queue is full, and therefore the frame we're rendering is always destined to be at the back of the queue, and so will not be displayed straight away on the next vsync as there are likely buffers before it in the queue.
In contrast rendering-on-demand typically happens much less frequently than the display update rate, so typically the BufferQueues for those views are empty, and therefore any updates pushed into those queues will be grabbed by SurfaceFlinger on the very next vsync.
So here's the question: How can I set up a continuous renderer, but with minimal lag? The goal is that the buffer queue is empty at the start of each vsync, I render my content in under 16ms, push it to the queue (buffer count = 1), and it is then consumed by SurfaceFlinger on the next vsync (buffer count = 0), repeat. The number of Buffers in the queue can be seen in systrace, so the goal is to have this alternate between 0 and 1.
The document I mention above introduces Choreographer as a way to get callbacks on each vsync. However I'm not convinced that is enough to be able to achieve the minimal lag behaviour I'm after. I have tested doing a requestRender() on a vsync callback with a very minimal onDrawFrame() and it does indeed exhibit the 0/1 buffer count behaviour. However what if SurfaceFlinger isn't able to do all of its work within a single frame period (perhaps a notification pops in or whatever)? In that case I expect my renderer will happily be producing 1 frame per vsync, but the consumer end of that BufferQueue has dropped a frame. Result: we're now alternating between 1 and 2 buffers in our queue, and we've gained a frame of lag between doing the rendering and seeing the frame.
The document appears to suggest looking at the time offset between the reported vsync time and when the callback is run. I can see how that can help if your callback is delivered late due to your main thread due to a layout pass or something. However I don't think that would allow detecting of SurfaceFlinger skipping a beat and failing to consume a frame. Is there any way the app can work out that SurfaceFlinger has dropped a frame? It also seems like inability to tell the length of the queue breaks the idea of using the vsync time for game-state updates, as there's an unknown number of frames in the queue before the one you're rendering will actually be displayed.
Reducing the maximum length of the queue and relying on the back-pressure would be one way to achieve this, but I don't think there's an API to set the maximum number of buffers in the GLSurfaceView BufferQueue?
Great question.
Quick bit of background for anyone else reading this:
The goal here is to minimize the display latency, i.e. the time between when the app renders a frame and when the display panel lights up the pixels. If you're just throwing content at the screen, it doesn't matter, because the user can't tell the difference. If you're responding to touch input, though, every frame of latency makes your app feel just a bit less responsive.
The problem is similar to A/V sync, where you need audio associated with a frame to come out the speaker as the video frame is being displayed on screen. In that case, the overall latency doesn't matter so long as its consistently equal on both audio and video outputs. This faces very similar problems though, because you'll lose sync if SurfaceFlinger stalls and your video is consistently being displayed one frame later.
SurfaceFlinger runs at elevated priority, and does relatively little work, so isn't likely to miss a beat on its own... but it can happen. Also, it is compositing frames from multiple sources, some of which uses fences to signal asynchronous completion. If an on-time video frame is composed with OpenGL output, and the GLES rendering hasn't completed when the deadline hits, the whole composition will be postponed to the next VSYNC.
The desire to minimize latency was strong enough that the Android KitKat (4.4) release introduced the "DispSync" feature in SurfaceFlinger, which shave half a frame of latency off the usual two-frame delay. (This is briefly mentioned in the graphics architecture doc, but it's not in widespread use.)
So that's the situation. In the past this was less of an issue for video, because 30fps video updates every-other frame. Hiccups work themselves out naturally because we're not trying to keep the queue full. We're starting to see 48Hz and 60Hz video though, so this matters more.
The question is, how do we detect if the frames we send to SurfaceFlinger are being displayed as soon as possible, or are spending an extra frame waiting behind a buffer we sent previously?
The first part of the answer is: you can't. There is no status query or callback on SurfaceFlinger that will tell you what its state is. In theory you could query the BufferQueue itself, but that won't necessarily tell you what you need to know.
The problem with queries and callbacks is that they can't tell you what the state is, only what the state was. By the time the app receives the information and acts on it, the situation may be completely different. The app will be running at normal priority, so it's subject to delays.
For A/V sync it's slightly more complicated, because the app can't know the display characteristics. For example, some displays have "smart panels" that have memory built in to them. (If what's on the screen doesn't update often, you can save a lot of power by not having the panel scan the pixels across the memory bus 60x per second.) These can add an additional frame of latency that must be accounted for.
The solution Android is moving toward for A/V sync is to have the app tell SurfaceFlinger when it wants the frame to be displayed. If SurfaceFlinger misses the deadline, it drops the frame. This was added experimentally in 4.4, though it's not really intended to be used until the next release (it should work well enough in "L preview", though I don't know if that includes all of the pieces required to use it fully).
The way an app uses this is to call the eglPresentationTimeANDROID() extension before eglSwapBuffers(). The argument to the function is the desired presentation time, in nanoseconds, using the same timebase as Choreographer (specifically, Linux CLOCK_MONOTONIC). So for each frame, you take the timestamp you got from the Choreographer, add the desired number of frames multiplied by the approximate refresh rate (which you can get by querying the Display object -- see MiscUtils#getDisplayRefreshNsec()), and pass it to EGL. When you swap buffers, the desired presentation time is passed along with the buffer.
Recall that SurfaceFlinger wakes up once per VSYNC, looks at the collection of pending buffers, and delivers a set to the display hardware via Hardware Composer. If you request display at time T, and SurfaceFlinger believes that a frame passed to the display hardware will display at time T-1 or earlier, the frame will be held (and the previous frame re-shown). If the frame will appear at time T, it will be sent to the display. If the frame will appear at time T+1 or later (i.e. it will miss its deadline), and there's another frame behind it in the queue that is scheduled for a later time (e.g. the frame intended for time T+1), then the frame intended for time T will be dropped.
The solution doesn't perfectly suit your problem. For A/V sync, you need constant latency, not minimum latency. If you look at Grafika's "scheduled swap" activity you can find some code that uses eglPresentationTimeANDROID() in a way similar to what a video player would do. (In its current state it's little more than a "tone generator" for creating systrace output, but the basic pieces are there.) The strategy there is to render a few frames ahead, so SurfaceFlinger never runs dry, but that's exactly wrong for your app.
The presentation-time mechanism does, however, provide a way to drop frames rather than letting them back up. If you happen to know that there are two frames of latency between the time reported by Choreographer and the time when your frame can be displayed, you can use this feature to ensure that frames will be dropped rather than queued if they are too far in the past. The Grafika activity allows you to set the frame rate and requested latency, and then view the results in systrace.
It would be helpful for an app to know how many frames of latency SurfaceFlinger actually has, but there isn't a query for that. (This is somewhat awkward to deal with anyway, as "smart panels" can change modes, thereby changing the display latency; but unless you're working on A/V sync, all you really care about is minimizing the SurfaceFlinger latency.) It's reasonably safe to assume two frames on 4.3+. If it's not two frames, you may have suboptimal performance, but the net effect will be no worse than you would get if you didn't set the presentation time at all.
You could try setting the desired presentation time equal to the Choreographer timestamp; a timestamp in the recent past means "show ASAP". This ensures minimum latency, but can backfire on smoothness. SurfaceFlinger has the two-frame delay because it gives everything in the system enough time to get work done. If your workload is uneven, you'll wobble between single-frame and double-frame latency, and the output will look janky at the transitions. (This was a concern for DispSync, which reduces the total time to 1.5 frames.)
I don't remember when the eglPresentationTimeANDROID() function was added, but on older releases it should be a no-op.
Bottom line: for 'L', and to some extent 4.4, you should be able to get the behavior you want using the EGL extension with two frames of latency. On earlier releases there's no help from the system. If you want to make sure there isn't a buffer in your way, you can deliberately drop a frame every so often to let the buffer queue drain.
这篇关于最小化的Android GLSurfaceView滞后的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
