低延迟（<2s）实时视频流HTML5解决方案？ [英] Low latency (&lt; 2s) live video streaming HTML5 solutions?

问题描述

目前在Flash + RTMP中，我有一个实时视频流与<我已经尝试过使用MPEG-DASH，它似乎是流媒体的新的行业标准，但是它的缺点是5秒延迟，最好我可以从它挤压。

对于上下文，我试图让用户来控制它们可以在流中看到的物理对象，所以任何超过几秒钟延迟会导致令人沮丧的体验。

是否还有其他技术，还是真的没有用于直播的低延迟html5解决方案？

技术和要求 b
$ b

WebRTC是唯一真正面向低延迟的基于网络的技术组合。它专为视频会议而设计。编解码器针对低质量进行了优化。比特率通常是可变的，选择稳定的连接而不是质量。

但是，对于所有用户而言，您不一定需要这种低延迟优化。事实上，根据我的需求，每个人的低延迟都会损害用户体验。虽然控制机器人的用户肯定需要低延迟的视频，以便他们可以合理地控制它，但没有控制权的用户不具备此要求，而是可以选择可靠的高质量视频。

如何设置

< h2>机内控制用户连接

控制机器人的用户将加载一个页面，该页面利用一些WebRTC组件连接到摄像机和控制服务器。为了方便WebRTC连接，您需要某种STUN服务器。要绕过NAT和其他防火墙限制，您可能需要TURN服务器。这些都通常构建在基于Node.js的WebRTC框架中。

凸轮/控制服务器还需要通过WebRTC连接。老实说，最简单的方法是使您的控制应用程序在某种程度上基于Web。由于您已经在使用Node.js，请查看 NW.js 或 Electron 。两者都可以利用已经在WebKit中构建的WebRTC功能，同时还可以灵活地随意使用Node.js。

控制中用户和cam / control服务器将通过WebRTC（或TURN服务器，如果需要）建立对等连接。从那里，您将需要打开媒体渠道和数据渠道。数据端可用于发送您的机器人命令。当然，媒体频道将用于将低延迟视频流发送回控制内用户。

同样，重要的是要注意，视频会被发回将会针对延迟进行优化，而不是质量。这种连接还可确保对您的命令做出快速响应。

查看用户的视频

用户只是查看流而不是控制机器人可以使用正常的视频分发方法。使用现有的CDN和转码服务对您来说非常重要，因为您将有10k-15k人在观看该视频流。有了这么多用户，你可能会想要在几种不同的编解码器中使用你的视频，当然还有一大堆比特率。通过 DASH 或 HLS 目前最容易使用，并且释放您的Flash要求。

您可能还希望将您的信息流发送到社交媒体服务。这是为什么从高质量HD流开始很重要的另一个原因。这些服务会再次转码您的视频，降低质量。如果您先从优质开始，那么最终会获得更好的质量。

元数据（聊天，控制信号等）

从您的需求中不清楚您需要什么样的元数据，但对于基于小型消息的数据，您可以使用网络套接字库，例如Socket.IO。随着您扩展到几个实例，您可以使用pub / sub（例如 Redis ）在整个服务器。

要将元数据同步到视频，具体取决于元数据中的内容以及同步要求。一般而言，您可以假设源视频和客户端之间会有合理但不可预测的延迟。毕竟，你无法控制缓冲时间。每个设备是不同的，每个连接变量。您可以假设的是，回放将从客户端下载的第一个分段开始。换句话说，如果客户开始缓冲视频并在2秒钟后开始播放视频，则该视频比第一次请求的时间晚2秒。

检测何时播放实际上开始客户端 可能。由于服务器知道视频发送到客户端的时间戳，因此它可以通知客户端其相对于视频播放开始的偏移量。由于您可能会使用DASH或HLS，因此无论如何您都需要使用带有AJAX的MCE来获取数据，因此您可以使用响应标头在段响应中指示段开始的时间戳。客户端然后可以自己同步。让我一步一步分解：

1. 客户端开始从应用服务器接收元数据消息。


2. 客户端向CDN请求第一个视频片段。


3. CDN服务器回复视频片段。在响应标题中， Date：标题可以指示段开始的确切日期/时间。
   
4. 客户端读取响应日期：标题（比方说 2016-06-01 20:31:00 ）。客户端会继续缓存这些段。


5. 客户端开始缓冲/回放。


6. 开始播放。客户端可以在播放器上检测到这种状态变化，并且知道视频播放器上的 00:00:00 是真实的 2016-06-01 20:31 ：00 。


7. 客户端显示与视频同步的元数据，放弃之前的任何消息并缓存任何未来时间。

这应该满足您的需求，并为您提供灵活性，以随时随地为视频做任何事情。

< h1>为什么不[魔术技术 - 这里]？

• 当你选择低延迟时，你会失去质量。质量来自可用带宽。带宽效率来自于在编码时能够缓冲和优化整个图像序列。如果你想要完美的质量（每张图像无损），你需要一吨（每个观众gigabites）的带宽。这就是为什么我们有这些有损编解码器的原因。


• 因为大多数观看者实际上并不需要低延迟，所以最好优化质量对于需要低延迟的15,000个用户中的2个用户，我们可以优化它们的低延迟。他们将获得不合格的视频质量，但将能够主动控制机器人，这真是太棒了！ 总是要记住，互联网是一个敌对的地方，没有任何工作可以完成， 。系统资源和带宽不断变化。这实际上是为什么WebRTC自动调整（尽可能合理）以适应不断变化的情况。
不是所有的连接都能满足低延迟的要求。这就是为什么每个低延迟连接都会出现退出的原因。互联网是分组交换，而不是电路交换。没有真正的专用带宽可用。

• 拥有一个大缓冲区（几秒钟）可以让客户端经受短暂的连接损失。这就是为什么创建具有反跳过缓冲区的CD播放器并且销售非常好的原因。如果视频正常工作，这15,000个用户的用户体验会更好。他们不必知道他们比主流落后5-10秒，但他们肯定会知道视频是否每隔一秒就消失一次。

每种方法都有权衡。我认为我在此概述的内容将这些问题分开，并为您提供每个领域的最佳权衡。请随时要求澄清或在评论中提出后续问题。

With Chrome disabling Flash by default very soon I need to start looking into flash/rtmp html5 replacement solutions.

Currently with Flash + RTMP I have a live video stream with < 1-2 second delay.

I've experimented with MPEG-DASH which seems to be the new industry standard for streaming but that came up short with 5 second delay being the best I could squeeze from it.

For context, I am trying to allow user's to control physical objects they can see on the stream, so anything above a couple of seconds of delay leads to a frustrating experience.

Are there any other techniques, or is there really no low latency html5 solutions for live streaming yet?

解决方案

Technologies and Requirements

The only web-based technology set really geared toward low latency is WebRTC. It's built for video conferencing. Codecs are tuned for low latency over quality. Bitrates are usually variable, opting for a stable connection over quality.

However, you don't necessarily need this low latency optimization for all of your users. In fact, from what I can gather on your requirements, low latency for everyone will hurt the user experience. While your users in control of the robot definitely need low latency video so they can reasonably control it, the users not in control don't have this requirement and can instead opt for reliable higher quality video.

How to Set it Up

In-Control Users to Robot Connection

Users controlling the robot will load a page that utilizes some WebRTC components for connecting to the camera and control server. To facilitate WebRTC connections, you need some sort of STUN server. To get around NAT and other firewall restrictions, you may need a TURN server. Both of these are usually built into Node.js-based WebRTC frameworks.

The cam/control server will also need to connect via WebRTC. Honestly, the easiest way to do this is to make your controlling application somewhat web based. Since you're using Node.js already, check out NW.js or Electron. Both can take advantage of the WebRTC capabilities already built in WebKit, while still giving you the flexibility to do whatever you'd like with Node.js.

The in-control users and the cam/control server will make a peer-to-peer connection via WebRTC (or TURN server if required). From there, you'll want to open up a media channel as well as a data channel. The data side can be used to send your robot commands. The media channel will of course be used for the low latency video stream being sent back to the in-control users.

Again, it's important to note that the video that will be sent back will be optimized for latency, not quality. This sort of connection also ensures a fast response to your commands.

Video for Viewing Users

Users that are simply viewing the stream and not controlling the robot can use normal video distribution methods. It is actually very important for you to use an existing CDN and transcoding services, since you will have 10k-15k people watching the stream. With that many users, you're probably going to want your video in a couple different codecs, and certainly a whole array of bitrates. Distribution with DASH or HLS is easiest to work with at the moment, and frees you of Flash requirements.

You will probably also want to send your stream to social media services. This is another reason why it's important to start with a high quality HD stream. Those services will transcode your video again, reducing quality. If you start with good quality first, you'll end up with better quality in the end.

Metadata (chat, control signals, etc.)

It isn't clear from your requirements what sort of metadata you need, but for small message-based data, you can use a web socket library, such as Socket.IO. As you scale this up to a few instances, you can use pub/sub, such as Redis, to distribution messaging throughout the servers.

To synchronize the metadata to the video depends a bit on what's in that metadata and what the synchronization requirement is, specifically. Generally speaking, you can assume that there will be a reasonable but unpredictable delay between the source video and the clients. After all, you cannot control how long they will buffer. Each device is different, each connection variable. What you can assume is that playback will begin with the first segment the client downloads. In other words, if a client starts buffering a video and begins playing it 2 seconds later, the video is 2 seconds behind from when the first request was made.

Detecting when playback actually begins client-side is possible. Since the server knows the timestamp for which video was sent to the client, it can inform the client of its offset relative to the beginning of video playback. Since you'll probably be using DASH or HLS and you need to use MCE with AJAX to get the data anyway, you can use the response headers in the segment response to indicate the timestamp for the beginning the segment. The client can then synchronize itself. Let me break this down step-by-step:

1. Client starts receiving metadata messages from application server.
2. Client requests the first video segment from the CDN.
3. CDN server replies with video segment. In the response headers, the Date: header can indicate the exact date/time for the start of the segment.
4. Client reads the response Date: header (let's say 2016-06-01 20:31:00). Client continues buffering the segments.
5. Client starts buffering/playback as normal.
6. Playback starts. Client can detect this state change on the player and knows that 00:00:00 on the video player is actualy 2016-06-01 20:31:00.
7. Client displays metadata synchronized with the video, dropping any messages from previous times and buffering any for future times.

This should meet your needs and give you the flexibility to do whatever you need to with your video going forward.

Why not [magic-technology-here]?

• When you choose low latency, you lose quality. Quality comes from available bandwidth. Bandwidth efficiency comes from being able to buffer and optimize entire sequences of images when encoding. If you wanted perfect quality (lossless for each image) you would need a ton (gigabites per viewer) of bandwidth. That's why we have these lossy codecs to begin with.
• Since you don't actually need low latency for most of your viewers, it's better to optimize for quality for them.
• For the 2 users out of 15,000 that do need low latency, we can optimize for low latency for them. They will get substandard video quality, but will be able to actively control a robot, which is awesome!
• Always remember that the internet is a hostile place where nothing works quite as well as it should. System resources and bandwidth are constantly variable. That's actually why WebRTC auto-adjusts (as best as reasonable) to changing conditions.
• Not all connections can keep up with low latency requirements. That's why every single low latency connection will experience drop-outs. The internet is packet-switched, not circuit-switched. There is no real dedicated bandwidth available.
• Having a large buffer (a couple seconds) allows clients to survive momentary losses of connections. It's why CD players with anti-skip buffers were created, and sold very well. It's a far better user experience for those 15,000 users if the video works correctly. They don't have to know that they are 5-10 seconds behind the main stream, but they will definitely know if the video drops out every other second.

There are tradeoffs in every approach. I think what I have outlined here separates the concerns and gives you the best tradeoffs in each area. Please feel free to ask for clarification or ask follow-up questions in the comments.

这篇关于低延迟（<2s）实时视频流HTML5解决方案？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！