使用带有socket.io的媒体源无法从媒体记录器跳入流 [英] Unable jump into stream from media recorder using media source with socket.io

查看:147
本文介绍了使用带有socket.io的媒体源无法从媒体记录器跳入流的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

以下代码在先加载视频观察客户端然后再加载网络摄像头客户端时可以正常工作,但是,如果切换顺序或以任何方式中断流(例如,通过刷新任一客户端来刷新流),则以下代码将正常工作失败,媒体源将其就绪状态更改为关闭.

The following code works when the video observing client is loaded first then the webcam client is loaded second it works flawlessly, however, if the order is switched or in any way the stream is interrupted for example by refreshing either client the stream will fail and the Media Source will change its ready state to closed.

我的假设是,在开始时接收的视频需要用于启动的初始化标头,并且由于在中途读取流,因此永远不会获得所述初始化标头.我不确定如何将这样的标头添加到webm文件中.

My assumption is that the video being received on start needs initialization headers for starting and since the stream is being read midstream it never gets said initialization headers. I am unsure of how to even add such header to the webm file.

我试图更改没有执行任何操作的源缓冲区上的序列模式.我已经尝试过重新启动录像机,并且可以正常工作,但是我的最终计划是拥有多个观察客户端,并且每次重新连接时重新启动录像机都不是最佳选择.

I have tried to change the sequence mode on the source buffer which did nothing. I have tried restarting the video recorder and that works, but my final plan is to have multiple observing clients and the video recorder restarting on every reconnection is not optimal.

相机客户端

main();
function main() {
    if (hasGetUserMedia()) {
        const constraints = {
            video: {
                facingMode: 'environment',
                frameRate: {
                    ideal: 10,
                    max: 15
                }
            },
            audio: true
        };

        navigator.mediaDevices.getUserMedia(constraints).
        then(stream => {
            setupRecorder(stream);
        });
    }
}

function setupRecorder(stream) {
    let mediaRecorder = new MediaRecorder(stream, {
        mimeType: 'video/webm; codecs="opus, vp9"'
    });

    mediaRecorder.ondataavailable = e => {
        var blob = e.data;
        socket.emit('video', blob);
    }

    mediaRecorder.start(500);
}

服务器只是广播收到的任何内容

The server just broadcasts whatever is received

观察客户

var sourceBuffer;
var queue = [];
var mediaSource = new MediaSource();
mediaSource.addEventListener('sourceopen', sourceOpen, false);
main();

socket.on('stream', data => {
    if (mediaSource.readyState == "open") {
        if (sourceBuffer.updating || queue.length > 0) {
            queue.push(data.video);
        } else {
            sourceBuffer.appendBuffer(data.video);
        }
    }
});

function main() {
    videoElement = document.querySelector('#video');
    videoElement.src = URL.createObjectURL(mediaSource);
}

function sourceOpen(e) {
    console.log('open');
    sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="opus, vp9"');
    sourceBuffer.addEventListener('updateend', () => {
        console.log(sourceBuffer.updating, mediaSource.readyState);

        if (queue.length > 0 && !sourceBuffer.updating) {
            sourceBuffer.appendBuffer(queue.shift());
        }
    });
}

因此,实际上,代码只是以一种不正确的方式工作,因此套接字发送服务器没有错.它与MediaRecorder或MediaSource有关.

So the code, in fact, works just in a way that is not correct so nothing is wrong with the server of socket sending. It either has something to do with the MediaRecorder or MediaSource.

推荐答案

我的假设是,在开始时接收的视频需要用于启动的初始化标头,并且由于在流的中间读取流,因此永远不会获得所述初始化标头.

My assumption is that the video being received on start needs initialization headers for starting and since the stream is being read midstream it never gets said initialization headers.

正确!

要解决此问题,您需要对WebM格式有所了解. WebM只是Matroska(MKV)的子集. Matroska 是用于在EBML中存储媒体的架构的规范. EBML 是可以具有任意块的二进制文件格式.可以将其视为二进制XML.

To solve this, you need to know a bit about the WebM format. WebM is just a subset of Matroska (MKV). Matroska is a specification of a schema for storing media in EBML. EBML is a binary file format that can have arbitrary blocks. Think of it like a binary XML.

这意味着您可以使用 EBML Viewer EBML Viewer 检查WebM文件,并参考Matroska规格以了解发生了什么.例如:

What this means is that you can use tools like EBML Viewer to inspect WebM files, and reference the Matroska specs to understand what's going on. For example:

这是对预先记录的WebM文件的检查.它将在浏览器中正常播放.您会注意到有些元素是嵌套的.

This is the inspection of a WebM file that was pre-recorded. It will play fine in browsers. You'll note that there are elements that are nested.

每个WebM文件中都有两个顶级元素. EBML(定义此二进制文件)和Segment(包含之后的所有内容).

There are two top-level elements found in every WebM file. EBML, which defines this binary file, and Segment which contains everything after.

Segment中,有几个对您很重要的元素. Tracks是其中之一.您会注意到,此文件有两条轨道,一条在Opus中用于音频,一条在VP9中用于视频.另一个重要的块是Info,其中包含有关时间刻度的信息以及有关多路复用器的一些元数据.

Within Segment there are a couple elements that matter to you. One of which is Tracks. You'll note that this file has two tracks, one for audio in Opus, and one for video in VP9. The other important block is Info, which contains information about the timescale and some metadata about the muxer.

在所有这些元数据之后,您会找到ClusterClusterCluster等.在这些位置,您可以剪切WebM流,前提是每个从关键帧开始.

After all of that metadata, you find Cluster, Cluster, Cluster, etc. These are the places in which you can cut a WebM stream, provided that each Cluster begins with a keyframe.

换句话说,您的代码应执行以下操作:

In other words, your code should do the following:

  • 将第一个Cluster之前的所有数据保存为初始化数据".​​
  • 之后在Cluster上拆分.
  • Save all data before the first Cluster as "initialization data".
  • Split on Cluster after that.

播放时:

  • 首先使用先前保存的初始化数据"进行加载.
  • 此后,开始在Cluster s中加载,从流中的任意位置开始.
  • Use the previously-saved "initialization data" as the first thing you load in.
  • Start loading in Clusters after that, starting wherever you want in the stream.

现在,谁需要一个关键帧位很重要.据我所知,无法配置MediaRecorder来做到这一点,浏览器对此特别挑剔.至少,您必须重新混合服务器端...您甚至可能需要重新编码.另请参阅:编码FFMPEG到MPEG-DASH –或带有关键帧群集的WebM –用于MediaSource API

Now, that who cluster-needs-a-keyframe bit is important. As far as I know, there's no way to configure MediaRecorder to do this, and browsers are particularly picky about this. At a minimum, you'll have to remux server-side... you might even need to re-encode. See also: Encoding FFMPEG to MPEG-DASH – or WebM with Keyframe Clusters – for MediaSource API

将媒体源与socket.io一起使用

using media source with socket.io

我应该指出,您甚至不需要MediaSource.您绝对不需要Socket.IO.就像通过普通HTTP流输出此数据一样简单.可直接将其加载到<video>元素中. (如果需要其他控制,请务必使用MediaSource,但这不是必需的.)

I should point out that you don't even need MediaSource for this. You definitely don't need Socket.IO. It can be as simple as outputting this data over a normal HTTP stream. This is loadable directly in a <video> element. (By all means, use MediaSource if you want additional control, but it isn't necessary.)

这篇关于使用带有socket.io的媒体源无法从媒体记录器跳入流的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆