寻找速度最快的视频编码器，将网络摄像头流传输到 ipad 的延迟最少 [英] Look for fastest video encoder with least lag to stream webcam streaming to ipad

问题描述

我正在寻找对可在 html5 视频标签中查看的网络摄像头流进行编码的最快方法.我正在使用 Pandaboard:http://www.digikey.com/product-highlights/us/en/texas-instruments-pandaboard/686#tabs-2 用于硬件.可以使用gstreamer、cvlc、ffmpeg.我将使用它来驱动机器人，因此视频流中需要最少的延迟.质量不必很好，也不需要音频.此外，这仅适用于一个客户端，因此带宽不是问题.迄今为止最好的解决方案是使用 ffmpeg 和 mpjpeg 给我大约 1 秒的延迟.有什么更好的吗?

I'm looking for the fastest way to encode a webcam stream that will be viewable in a html5 video tag. I'm using a Pandaboard: http://www.digikey.com/product-highlights/us/en/texas-instruments-pandaboard/686#tabs-2 for the hardware. Can use gstreamer, cvlc, ffmpeg. I'll be using it to drive a robot, so need the least amount of lag in the video stream. Quality doesn't have to be great and it doesn't need audio. Also, this is only for one client so bandwidth isn't an issue. The best solution so far is using ffmpeg with a mpjpeg gives me around 1 sec delay. Anything better?

推荐答案

这个问题我已经被问过很多次了，所以我会尽量笼统地回答这个问题，而不仅仅是针对 mjpeg.在系统中获得非常低的延迟需要一些系统工程工作以及对组件的了解.

I have been asked this many times so I will try and answer this a bit generically and not just for mjpeg. Getting very low delays in a system requires a bit of system engineering effort and also understanding of the components.

我能想到的一些简单的顶级调整是:

Some simple top level tweaks I can think of are:

确保编解码器配置为最低延迟.编解码器(尤其是嵌入式系统编解码器)将具有低延迟配置.启用它.如果您使用的是 H.264，它是最有用的.大多数人没有意识到按照标准要求，H.264 解码器需要在显示之前缓冲帧.Qcif 最多 16 帧，720p 最多 5 帧.这是获取第一帧的大量延迟.如果您不使用 H.264，请确保您没有启用 B 图片.这增加了获取第一张图片的延迟.

Ensure the codec is configured for the lowest delay. Codecs will have (especially embedded system codecs) a low delay configuration. Enable it. If you are using H.264 it's most useful. Most people don't realize that by standard requirements H.264 decoders need to buffer frames before displaying it. This can be upto 16 for Qcif and upto 5 frames for 720p. That is a lot of delay in getting the first frame out. If you do not use H.264 still ensure you do not have B pictures enabled. This adds delay to getting the first picture out.

由于您使用的是 mjpeg，因此我认为这对您不太适用.

Since you are using mjpeg, I don't think this is applicable to you much.

编码器也会有速率控制延迟.(称为初始化延迟或 vbv buf 大小).将其设置为可提供可接受质量的最小值.这也将减少延迟.将此视为编码器和解码器之间的比特流缓冲区.如果您使用的是 x264，那将是 vbv 缓冲区大小.

Encoders will also have a rate control delay. (Called init delay or vbv buf size). Set it to the smallest value that gives you acceptable quality. That will also reduce the delay. Think of this as the bitstream buffer between encoder and decoder. If you are using x264 that would be the vbv buffer size.

一些简单的其他配置:使用尽可能少的 I 图片(大的内部周期).我的图片很大，增加了通过网络发送的延迟.这在端到端延迟在 1 秒或更长范围内的系统中可能不是很明显，但是当您设计需要 100 毫秒或更短的端到端延迟的系统时，这和其他几个方面就会发挥作用.还要确保您使用的是低延迟音频编解码器 aac-lc(而不是 heaac).

Some simple other configurations: Use as few I pictures as possible (large intra period). I pictures are huge and add to the delay to send over the network. This may not be very visible in systems where end to end delay is in the range of 1 second or more but when you are designing systems that need end to end delay of 100ms or less, this and several other aspects come into play. Also ensure you are using a low latency audio codec aac-lc (and not heaac).

在您的情况下，要降低延迟，我建议您远离 mjpeg 并至少使用没有 B 图片的 mpeg4(简单配置文件)，或者最好是 H.264 基线配置文件(x264 提供零延迟选项).您将获得较低延迟的一个简单原因是，您将获得较低的编码后比特率以发送数据，并且您可以达到全帧率.如果您必须坚持使用 mjpeg，那么您几乎可以在没有使用开源组件的编解码器和系统提供更多高级功能支持的情况下获得.

In your case to get to lower latencies I would suggest moving away from mjpeg and use at least mpeg4 without B pictures (Simple profile) or best is H.264 baseline profile (x264 gives a zerolatency option). The simple reason you will get lower latency is that you will get lower bitrate post encoding to send the data out and you can go to full framerate. If you must stick to mjpeg you have close to what you can get without more advanced features support from the codec and system using the open source components as is.

另一方面是将内容传输到显示单元.如果您可以使用 udp，与 tcp 相比，它将大大减少延迟，尽管有时会根据网络状况而有损.您提到了 html5 视频.我很好奇你是如何对 html5 视频标签进行直播的.

Another aspect is the transmission of the content to the display unit. If you can use udp it will reduce latency quite a lot compared to tcp, though it can be lossy at times depending on network conditions. You have mentioned html5 video. I am curious as to how you are doing live streaming to a html5 video tag.

还有其他方面可以调整，我会把它们放在高级类别中，需要系统工程师尝试各种方法

There are other aspects that can also be tweaked which I would put in the advanced category and requires the system engineer to try various things out

操作系统中的网络缓冲是什么?出于性能原因，操作系统还在发送数据之前缓冲数据.调整它以获得性能和速度之间的良好平衡.

What is the network buffering in the OS? The OS also buffers data before sending it out for performance reasons. Tweak this to get a good balance between performance and speed.

您使用的是 CR 还是 VBR 编码?虽然 CBR 非常适合低抖动，但如果编解码器提供了上限 vbr，您也可以使用它.

Are you using CR or VBR encoding? While CBR is great for low jitter you can also use capped vbr if the codec provides it.

您的解码器可以开始解码部分帧吗?因此，在将数据提供给解码器之前，您不必担心对数据进行分帧.保持尽快将数据推送到解码器即可.

Can your decoder start decoding partial frames? So you don't have to worry about framing the data before providing it to the decoder. Just keep pushing the data to the decoder as soon as possible.

你能做字段编码吗?将第一张图片从帧编码到输出的时间减半.

Can you do field encoding? Halves the time from frame encoding before getting the first picture out.

当切片可以立即通过网络发送时，您是否可以使用回调进行切片编码?

Can you do sliced encoding with callbacks whenever a slice is available to send over the network immediately?

在低于 100 毫秒的延迟系统中使用了我在上述所有工作中工作过的系统.某些功能在开源组件中可能不可用，但如果您确实需要它并且充满热情，您可以继续实施它们.

In sub 100 ms latency systems that I have worked in all of the above are used. Some of the features may not be available in open source components but if you really need it and are enthusiastic you could go ahead and implement them.

我意识到您无法对 ipad 流媒体解决方案执行上述很多操作，并且由于 hls 对您可以实现的延迟也存在限制.但我希望在您需要任何低延迟系统的其他情况下，它会被证明是有用的.

I realize you cannot do a lot of the above for a ipad streaming solution and there are limitations because of hls also to the latency you can achieve. But I hope it will prove useful in other cases when you need any low latency system.

这篇关于寻找速度最快的视频编码器，将网络摄像头流传输到 ipad 的延迟最少的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！