如何保持实时的MediaSource视频流同步？ [英] How to keep a live MediaSource video stream in-sync?
问题描述
我有一个服务器应用程序，它呈现30 FPS视频流，然后将其实时编码和复用到 WebM字节流。
I have a server application which renders a 30 FPS video stream then encodes and muxes it in real-time into a WebM Byte Stream.
在客户端，一个HTML5页面打开一个WebSocket到服务器，当接受连接时，它开始生成流。传递标头后，每个后续WebSocket框架都包含一个WebM SimpleBlock。关键帧每15帧出现一次，当发生这种情况时，会启动一个新的群集。
On the client side, an HTML5 page opens a WebSocket to the server, which starts generating the stream when connection is accepted. After the header is delivered, each subsequent WebSocket frame consists of a single WebM SimpleBlock. A keyframe occurs every 15 frames and when this happens a new Cluster is started.
客户端还会创建一个 MediaSource ，并在从WS接收帧时，将内容附加到其活动状态缓冲。 < video> 在追加第一帧后立即开始播放。
The client also creates a MediaSource, and on receiving a frame from the WS, appends the content to its active buffer. The <video> starts playback immediately after the first frame is appended.
一切运作良好。我唯一的问题是网络抖动导致播放位置在一段时间后偏离实际时间。我目前的解决方案是挂钩 updateend 事件，检查 video.currentTime 与时间码之间的差异传入的群集并手动更新 currentTime ，如果它超出可接受的范围。不幸的是，这会导致播放中出现明显的暂停和跳跃，这是相当不愉快的。
Everything works reasonably well. My only issue is that the network jitter causes the playback position to drift from the actual time after a while. My current solution is to hook into the updateend event, check the difference between the video.currentTime and the timecode on the incoming Cluster and manually update the currentTime if it falls outside an acceptable range. Unfortunately, this causes a noticeable pause and jump in the playback which is rather unpleasant.
解决方案也有点奇怪：我确切知道最新关键帧的位置，在将其传递到 currentTime 之前，我必须将其转换为整秒（根据W3C规范），其中浏览器可能必须绕过并找到最近的关键帧。
The solution also feels a bit odd: I know exactly where the latest keyframe is, yet I have to convert it into a whole second (as per the W3C spec) before I can pass it into currentTime, where the browser presumably has to then go around and find the nearest keyframe.
我的问题是：有没有办法让媒体元素始终寻找可用的最新关键帧，或保持播放时间与系统时钟同步时间？
My question is this: is there a way to tell the Media Element to always seek to the latest keyframe available, or keep the playback time synchronised with the system clock time?
推荐答案
网络抖动导致播放位置漂移
network jitter causes the playback position to drift
这不是你的问题。如果您在流中遇到丢失，那么在开始播放之前您没有足够的缓冲，并且回放只有一个适当大小的缓冲区，即使实时落后几秒（这是正常的）。
That's not your problem. If you are experiencing drop-outs in the stream, you aren't buffering enough before playback to begin with, and playback just has an appropriately sized buffer, even if a few seconds behind realtime (which is normal).
我当前的解决方案是挂钩updateend事件，检查video.currentTime和传入集群上的时间码之间的区别
My current solution is to hook into the updateend event, check the difference between the video.currentTime and the timecode on the incoming Cluster
这接近正确的方法。我建议你忽略传入集群的时间码，而是检查你的缓冲时间范围。您在WebM群集上收到的内容以及已解码的内容有两种不同的东西。
That's close to the correct method. I suggest you ignore the timecode of incoming cluster and instead inspect your buffered time ranges. What you've received on the WebM cluster, and what's been decoded are two different things.
不幸的是，这会导致明显的暂停和跳转在回放中相当不愉快。
Unfortunately, this causes a noticeable pause and jump in the playback which is rather unpleasant.
你还能怎么做？您可以跳转到实时，也可以提高播放速度以赶上实时。无论哪种方式，如果你想赶上实时，你必须及时跳过这一点。
How else would you do it? You can either jump to realtime, or you can increase playback speed to catch up to realtime. Either way, if you want to catch up to realtime, you have to skip in time to do that.
解决方案也有点感觉奇数：我确切地知道最新关键帧的位置
The solution also feels a bit odd: I know exactly where the latest keyframe is
您可以，但播放器不会解码该媒体。在任何情况下，关键帧都无关紧要......您可以寻找非关键帧位置。浏览器将根据需要在P / B帧之前进行解码。
You may, but the player doesn't until that media is decoded. In any case, keyframe is irrelevant... you can seek to non-keyframe locations. The browser will decode ahead of P/B-frames as required.
我必须将其转换为整秒（根据W3C） spec）之前我可以将它传递给currentTime
I have to convert it into a whole second (as per the W3C spec) before I can pass it into currentTime
这是完全错误的。 currentTime 被指定为 double 。 https：//www.w3。 org / TR / 2011 / WD-html5-20110113 / video.html＃dom-media-currenttime
That's totally false. The currentTime is specified as a double. https://www.w3.org/TR/2011/WD-html5-20110113/video.html#dom-media-currenttime
我的问题是这个：有没有办法告诉媒体元素总是寻找可用的最新关键帧，或者保持播放时间与系统时钟时间同步？
My question is this: is there a way to tell the Media Element to always seek to the latest keyframe available, or keep the playback time synchronised with the system clock time?
它将自动播放最后一个缓冲区。你不需要做任何事情。你正在通过确保媒体数据落在缓冲区中并将播放设置为合理的接近来完成工作。如果网络状况发生变化可以让你这么做，你总是可以向前推进它，但坦率地说，这听起来好像你只是破坏了代码和破坏缓冲策略。否则，回放将非常顺利。
It's going to play the last buffer automatically. You don't need to do anything. You're doing your job by ensuring media data lands in the buffer and setting playback as close to that as reasonable. You can always advance it forward if a network condition changes that allows you to do this, but frankly it sounds as if you just have broken code and a broken buffering strategy. Otherwise, playback would be simply smooth.
如果落后的追赶不会自动发生，也不应该。如果播放器因缓冲区耗尽而暂停，则需要重新建立缓冲区，然后才能恢复播放。这就是缓冲区的重点。
Catching up if fallen behind is not going to happen automatically, and nor should it. If the player pauses due to the buffer being drained, a buffer needs to be built back up again before playback can resume. That's the whole point of the buffer.
此外，您对系统时钟保持及时的期望不是一个好主意，也是不合理的。不同的设备具有不同的刷新率，将以不同的速率处理视频。只是点击播放并让它播放。如果您最终关闭几秒钟，请继续设置 currentTime ，但在此之前对您已缓冲的内容非常有信心。
Furthermore, your expectation of keeping anything in-time with the system clock is not a good idea and is unreasonable. Different devices have different refresh rates, will handle video at different rates. Just hit play and let it play. If you end up being several seconds off, go ahead and set currentTime, but be very confident of what you've buffered before doing so.
这篇关于如何保持实时的MediaSource视频流同步？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
