MFCreateFMPEG4MediaSink无法生成MSE兼容的MP4 [英] MFCreateFMPEG4MediaSink does not generate MSE-compatible MP4

问题描述

我正在尝试将H.264视频供稿流式传输到Web浏览器. Media Foundation用于编码分段的MPEG4流(启用了MFTranscodeContainerType_FMPEG4，MF_LOW_LATENCY和MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS的MFCreateFMPEG4MediaSink).然后，该流通过IMFByteStream连接到Web服务器.

I'm attempting to stream a H.264 video feed to a web browser. Media Foundation is used for encoding a fragmented MPEG4 stream (MFCreateFMPEG4MediaSink with MFTranscodeContainerType_FMPEG4, MF_LOW_LATENCY and MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS enabled). The stream is then connected to a web server through IMFByteStream.

由<video src=".."/>标签使用H.264视频时，其流传输可以正常工作.但是，由此产生的延迟约为2秒，对于所讨论的应用程序而言，这太高了.我怀疑客户端缓冲会导致大部分延迟.因此，我正在尝试使用媒体源扩展(MSE)对浏览器内流进行编程控制.但是，当通过MSE使用相同的MPEG4流时，Chrome浏览器确实会失败，并出现以下错误:

Streaming of the H.264 video works fine when it's being consumed by a <video src=".."/> tag. However, the resulting latency is ~2sec, which is too much for the application in question. My suspicion is that client-side buffering causes most of the latency. Therefore, I'm experimenting with Media Source Extensions (MSE) for programmatic control over the in-browser streaming. Chrome does, however, fail with the following error when consuming the same MPEG4 stream through MSE:

无法解析MP4:MSE不允许TFHD基本数据偏移.看 https://www.w3 .org/TR/mse-byte-stream-format-isobmff/#movie-fragment-relative-addressing

Failure parsing MP4: TFHD base-data-offset not allowed by MSE. See https://www.w3.org/TR/mse-byte-stream-format-isobmff/#movie-fragment-relative-addressing

MPEG4流中的moat/mdat片段的

mp4dump.这清楚地表明TFHD包含一个非法"的base data offset参数:

mp4dump of a moof/mdat fragment in the MPEG4 stream. This clearly shows that the TFHD contains an "illegal" base data offset parameter:

[moof] size=8+200
  [mfhd] size=12+4
    sequence number = 3
  [traf] size=8+176
    [tfhd] size=12+16, flags=1
      track ID = 1
      base data offset = 36690
    [trun] size=12+136, version=1, flags=f01
      sample count = 8
      data offset = 0
[mdat] size=8+1624

我正在使用在Win10版本1709(16299.309)上运行的Chrome 65.0.3325.181(正式版本)(32位).

I'm using Chrome 65.0.3325.181 (Official Build) (32-bit), running on Win10 version 1709 (16299.309).

是否可以使用Media Foundation生成与MSE兼容的H.264/MPEG4视频流?

Is there any way of generating a MSE-compatible H.264/MPEG4 video stream using Media Foundation?

基于 roman-r 建议，我设法通过拦截生成的MPEG4流并自行解决了该问题，执行以下修改:

Based on roman-r advise, I managed to fix the problem myself by intercepting the generated MPEG4 stream and perform the following modifications:

• 修改 Track Fragment Header Box (tfhd):
  • 删除base_data_offset参数(将流大小减小8个字节)
  • 设置default-base-is-moof标志

• Modify Track Fragment Header Box (tfhd):
  • remove base_data_offset parameter (reduces stream size by 8bytes)
  • set default-base-is-moof flag

• 设置baseMediaDecodeTime参数

• set baseMediaDecodeTime parameter

• 调整data_offset参数

• adjust data_offset parameter

字段说明记录在 https://www.iso.org/standard/68960 .html (免费下载).

The field descriptions are documented in https://www.iso.org/standard/68960.html (free download).

切换到基于MSE的视频流，将等待时间从〜2.0秒减少到0.7秒.不幸的是，这对于我来说仍然太多了.剩余延迟的主要来源似乎是由每个MP4片段中8个帧/样本的捆绑引起的.我不知道该如何解决.

Switching to MSE-based video streaming reduced the latency from ~2.0 to 0.7 sec. Unfortunately, this is still too much for my needs. The main source of remaining latency appear to be caused by the bundling of 8 frames/samples in each MP4 fragment. I don't know how to fix this.

https://github.com/forderud/AppWebStream 上有一个示例实现.

There's a sample implementation available on https://github.com/forderud/AppWebStream

推荐答案

提到的0.7秒延迟(在您的状态更新中)是由Media Foundation的MFTranscodeContainerType_FMPEG4容器所引起的，该容器收集并输出每个容器一对MP4 moof/mdat盒对中大约1/3秒(出于未知原因)的帧.这意味着您需要等待19帧，然后才能以60 FPS的速度从MFTranscodeContainerType_FMPEG4获得任何输出.

The mentioned 0.7 sec latency (in your Status Update) is caused by the Media Foundation's MFTranscodeContainerType_FMPEG4 containterizer which gathers and outputs each roughly 1/3 seconds (from unknown reason) of frames in one MP4 moof/mdat box pair. This means that you need to wait 19 frames before getting any output from MFTranscodeContainerType_FMPEG4 at 60 FPS.

要每帧输出单个MP4 moof/mdat，只需说谎MF_MT_FRAME_RATE为1 FPS(或高于1/3秒的任何值)即可.要以正确的速度播放视频，请使用Media Source Extensions的<video>.playbackRate或更新MP4流拦截器中的mvhd和mdhd框的timescale(即乘以实际FPS)以获取正确定时的MP4流.

To output single MP4 moof/mdat per each frame, simply lie that MF_MT_FRAME_RATE is 1 FPS (or anything higher than 1/3 sec). To play the video at the correct speed, use Media Source Extensions' <video>.playbackRate or rather update timescale (i.e. multiply by real FPS) of mvhd and mdhd boxes in your MP4 stream interceptor to get the correctly timed MP4 stream.

这样做，可以将等待时间压缩到20毫秒以下.当您在Unity(research)-> NvEnc-> MFTranscodeContainerType_FMPEG4-> WebSocket-> Chrome Media Source Extensions显示之类的链中的localhost上并排看到输出时，这几乎是无法识别的.

Doing that, the latency can be squeezed to under 20 ms. This is barely recognizable when you see the output side by side on localhost in chains such as Unity (research) -> NvEnc -> MFTranscodeContainerType_FMPEG4 -> WebSocket -> Chrome Media Source Extensions display.

请注意，MFTranscodeContainerType_FMPEG4仍然会引入1帧延迟(第一帧输入，无输出，第二帧输入，第一帧输出等)，因此在60 FPS时有20 ms的延迟.唯一的解决方案似乎是编写自己的FMPEG4容器化器.但这比拦截Media Foundation的MP4流要复杂一个数量级.

Note that MFTranscodeContainerType_FMPEG4 still introduces 1 frame delay (1st frame in, no output, 2nd frame in, 1st frame out, ...), hence the 20 ms latency at 60 FPS. The only solution to that seems to be writing own FMPEG4 containerizer. But that is order of magnitude more complex than intercepting of Media Foundation's MP4 streams.

这篇关于MFCreateFMPEG4MediaSink无法生成MSE兼容的MP4的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！