CanvasCaptureMediaStream/MediaRecorder 帧同步 [英] CanvasCaptureMediaStream / MediaRecorder Frame Synchronization

问题描述

当使用 CanvasCaptureMediaStream 和 MediaRecorder 时，有没有办法在每一帧上获取一个事件?

When using CanvasCaptureMediaStream and MediaRecorder, is there a way to get an event on each frame?

我需要的与 requestAnimationFrame() 没什么不同，但我需要它用于 CanvasCaptureMediaStream(和/或 MediaRecorder)而不是窗口.MediaRecorder 可能以与窗口不同的帧速率运行(可能以不规则可分的速率运行，例如 25 FPS 与 60 FPS)，所以我想以其帧速率而不是窗口的帧速率更新画布.

What I need is not unlike requestAnimationFrame(), but I need it for the CanvasCaptureMediaStream (and/or the MediaRecorder) and not the window. The MediaRecorder could be running at a different frame rate than the window (possibly at a not regularly divisible rate, such as 25 FPS vs 60 FPS), so I want to update the canvas at its frame rate rather than the window's.

推荐答案

此示例目前仅在Firefox 上完全有效，因为Chrome只在选项卡模糊时停止画布流... _{(可能与这个错误有关，但好吧，我的计时器似乎在工作，但没有在录音...)}

This example currently only fully works on FireFox, since chrome simply stops the canvas stream when the tab is blurred... _{(probably related to this bug, but well, my timer seems to be working but not the recording...)}

:它实际上现在只适用于 chrome，因为他们已经修复了 这个错误，但在 FF 中不再存在，因为 这个(由e10s引起).

: it actually now works only in chrome, since they have fixed this bug, but not anymore in FF because of this one (caused by e10s).

MediaStream 上似乎没有任何事件让您知道何时向其呈现帧，MediaRecorder 上也没有.

There doesn't seem to be any event on MediaStream letting you know when a frame has been rendered to it, neither on the MediaRecorder.

即使是 MediaStream 的 currentTime 属性(目前仅在 FF 中可用)似乎也没有随着 captureStream() 方法中传递的 fps 参数发生相应变化.

Even the currentTime property of the MediaStream (currently only available in FF) doesn't seem to be changing accordingly with the fps argument passed in the captureStream() method.

但您似乎想要的是一个可靠的计时器，当当前选项卡未聚焦时(即 rAF 会发生这种情况)不会丢失其频率.
幸运的是，WebAudio API 也有一个高精度计时器，基于硬件时钟，而不是屏幕刷新率.

But what you seem to want is a reliable timer, that won't loose its frequency when i.e the current tab is not focused (which happens for rAF).
Fortunately, the WebAudio API does also have an high precision timer, based on hardware clock, rather than on screen refresh rate.

所以我们可以提供一个替代的定时循环，即使在标签模糊时也能保持其频率.

So we can come with an alternative timed loop, able to keep its frequency even when the tab is blurred.

/*
	An alternative timing loop, based on AudioContext's clock

	@arg callback : a callback function 
		with the audioContext's currentTime passed as unique argument
	@arg frequency : float in ms;
	@returns : a stop function
	
*/
function audioTimerLoop(callback, frequency) {

  // AudioContext time parameters are in seconds
  var freq = frequency / 1000;

  var aCtx = new AudioContext();
  // Chrome needs our oscillator node to be attached to the destination
  // So we create a silent Gain Node
  var silence = aCtx.createGain();
  silence.gain.value = 0;
  silence.connect(aCtx.destination);

  onOSCend();

  var stopped = false;
  function onOSCend() {
    osc = aCtx.createOscillator();
    osc.onended = onOSCend;
    osc.connect(silence);
    osc.start(0);
    osc.stop(aCtx.currentTime + freq);
    callback(aCtx.currentTime);
    if (stopped) {
      osc.onended = function() {
        return;
      };
    }
  };
  // return a function to stop our loop
  return function() {
    stopped = true;
  };
}


function start() {

  // start our loop @25fps
  var stopAnim = audioTimerLoop(anim, 1000 / 25);
  // maximum stream rate set as 25 fps
  cStream = canvas.captureStream(25);

  let chunks = [];
  var recorder = new MediaRecorder(cStream);
  recorder.ondataavailable = e => chunks.push(e.data);
  recorder.onstop = e => {
    // we can stop our loop
    stopAnim();
    var url = URL.createObjectURL(new Blob(chunks));
    var v = document.createElement('video');
    v.src = url;
    v.controls = true;
    document.body.appendChild(v);
  }
  recorder.start();
  // stops the recorder in 20s, try to change tab during this time
  setTimeout(function() {
    recorder.stop();
  }, 20000)
}


// make something move on the canvas
var ctx = canvas.getContext('2d');
var x = 0;
function anim() {
  x = (x + 2) % (canvas.width + 100);
  ctx.fillStyle = 'ivory';
  ctx.fillRect(0, 0, canvas.width, canvas.height);
  ctx.fillStyle = 'red';
  ctx.fillRect(x - 50, 20, 50, 50)
};

btn.onclick = start;

<button id="btn">begin</button>
<canvas id="canvas" width="500" height="200"></canvas>

诺塔本:
在本例中，我将频率设置为 25fps，但我们可以将其设置为 60fps，即使在我的旧笔记本上，它似乎也能正常工作，至少有这么简单的动画.

Nota Bene :
In this example, I set the frequency to 25fps, but we can set it to 60fps and it seems to work correctly even on my old notebook, at least with such a simple animation.

这篇关于CanvasCaptureMediaStream/MediaRecorder 帧同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！