即使在慢速计算机上也可以使用 CanvasCaptureMediaStream 以恒定 fps 录制 [英] Record at constant fps with CanvasCaptureMediaStream even on slow computers

问题描述

我想从 HTML <canvas> 元素以特定帧速率录制视频.

I want to record a video from a HTML <canvas> element at a specific frame rate.

我将 CanvasCaptureMediaStream 与 canvas.captureStream(fps) 一起使用，并且还可以通过 const track = stream.getVideoTracks()[0] 访问视频轨道，所以我创建 track.requestFrame() 以通过 MediaRecorder 将其写入输出视频缓冲区.

I am using CanvasCaptureMediaStream with canvas.captureStream(fps) and also have access to the video track via const track = stream.getVideoTracks()[0] so I create track.requestFrame() to write it to the output video buffer via MediaRecorder.

我想一次精确地捕捉一帧，然后更改画布内容.更改画布内容可能需要一些时间(因为需要加载图像等).所以我无法实时捕捉画布.画布上的一些变化会在 500 毫秒内实时发生，因此这也需要调整为一次渲染一帧.

I want to precisely capture one frame at a time and then change the canvas content. Changing the canvas content can take some time (as images need to be loaded etc). So I can not capture the canvas in real-time. Some changes on the canvas would happen in 500ms real-time so this needs also to be adjusted to rendering one frame at the time.

推荐答案

MediaRecorder API 是用来记录 live-streams，做版本不是它的设计目的，它没有老实说，做得不好...

The MediaRecorder API is meant to record live-streams, doing edition is not what it was designed to do, and it doesn't do it very well to be honest...

MediaRecorder 本身没有帧率的概念，这通常由 MediaStreamTrack 定义.但是，CanvasCaptureStreamTrack 并没有真正说明它的帧速率是多少.
我们可以将参数传递给 HTMLCanvas.captureStream()，但这只是告诉我们每秒想要的最大帧数，它并不是真正的 fps 参数.
此外，即使我们停止在画布上绘图，录制器仍会继续实时延长录制视频的持续时间(我认为技术上只录制单个 long 帧，但在这种情况下).

The MediaRecorder itself has no concept of frame-rate, this is normally defined by the MediaStreamTrack. However, the CanvasCaptureStreamTrack doesn't really make it clear what its frame rate is.
We can pass a parameter to HTMLCanvas.captureStream(), but this only tells the max frames we want per seconds, it's not really an fps parameter.
Also, even if we stop drawing on the canvas, the recorder will still continue to extend the duration of the recorded video in real time (I think that technically only a single long frame is recorded though in this case).

所以...我们将不得不破解...

我们可以用 MediaRecorder 做的一件事是 pause() 和 resume() 它.
然后听起来很容易在进行长绘图操作之前暂停并在它完成后立即恢复?是的......而且也不是那么容易......
再一次，帧率是由 MediaStreamTrack 决定的，但是这个 MediaStreamTrack 是不能暂停的.
嗯，实际上有一种方法可以暂停一种特殊的 MediaStreamTrack，幸运的是我说的是 CanvasCaptureMediaStreamTracks.
当我们使用 0 参数调用我们的捕获流时，我们基本上可以手动控制何时将新帧添加到流中.
所以在这里我们可以将 MediaRecorder 和 MediaStreamTrack 同步到我们想要的任何帧速率.

One thing we can do with the MediaRecorder is to pause() and resume() it.
Then sounds quite easy to pause before doing the long drawing operation and to resume right after it's been made? Yes... and not that easy either...
Once again, the frame-rate is dictated by the MediaStreamTrack, but this MediaStreamTrack can not be paused.
Well, actually there is one way to pause a special kind of MediaStreamTrack, and luckily I'm talking about CanvasCaptureMediaStreamTracks.
When we do call our capture-stream with a parameter of 0, we are basically having manual control over when new frames are added to the stream.
So here we can synchronize both our MediaRecorder adn our MediaStreamTrack to whatever frame-rate we want.

基本的工作流程是

await the_long_drawing_task;
resumeTheRecorder();
writeTheFrameToStream(); // track.requestFrame();
await wait( time_per_frame );
pauseTheRecorder();

这样做，记录器仅在我们决定的每帧时间唤醒，并且在此期间将单个帧传递给 MediaStream，有效地模拟了 MediaRecorder 所关注的恒定 FPS 绘图.

Doing so, the recorder is awaken only the time per frame we decided, and a single frame is passed to the MediaStream during this time, effectively mocking a constant FPS drawing for what the MediaRecorder is concerned.

但与往常一样，在这个仍处于试验阶段的黑客行为带来了很多浏览器的怪异之处，以下演示实际上仅适用于当前的 Chrome...

But as always, hacks in this still experimental area come with a lot of browsers weirdness and the following demo actually only works in current Chrome...

无论出于何种原因，Firefox 生成的文件的帧数总是比请求的帧数多一倍，而且它偶尔还会在第一帧前面添加一个长...

For whatever reasons, Firefox will always generate files with twice the number of frames than what has been requested, and it will also occasionally prepend a long first frame...

另外需要注意的是，Chrome 有一个错误它将在绘图时更新画布流，即使我们使用 0 的 frameRequestRate 启动此流.因此，这意味着如果您在一切准备就绪之前开始绘图，或者如果在画布上绘图本身需要很长时间，那么我们的记录器将记录我们没有要求的半生不熟的帧.
为了解决这个错误，我们需要使用第二个画布，仅用于流式传输.我们将在该画布上做的就是绘制源画布，这将始终是一个足够快的操作.不要面对那个错误.

Also to be noted, Chrome has a bug where it will update the canvas stream at drawing, even though we initiated this stream with a frameRequestRate of 0. So this means that if you start drawing before everything is ready, or if the drawing on your canvas itself takes a long time, then our recorder will record half-baked frames that we didn't asked for.
To workaround this bug, we thus need to use a second canvas, used only for the streaming. All we'll do on that canvas is to drawImage the source one, which will always be a fast enough operation. to not face that bug.

class FrameByFrameCanvasRecorder {
  constructor(source_canvas, FPS = 30) {
  
    this.FPS = FPS;
    this.source = source_canvas;
    const canvas = this.canvas = source_canvas.cloneNode();
    const ctx = this.drawingContext = canvas.getContext('2d');

    // we need to draw something on our canvas
    ctx.drawImage(source_canvas, 0, 0);
    const stream = this.stream = canvas.captureStream(0);
    const track = this.track = stream.getVideoTracks()[0];
    // Firefox still uses a non-standard CanvasCaptureMediaStream
    // instead of CanvasCaptureMediaStreamTrack
    if (!track.requestFrame) {
      track.requestFrame = () => stream.requestFrame();
    }
    // prepare our MediaRecorder
    const rec = this.recorder = new MediaRecorder(stream);
    const chunks = this.chunks = [];
    rec.ondataavailable = (evt) => chunks.push(evt.data);
    rec.start();
    // we need to be in 'paused' state
    waitForEvent(rec, 'start')
      .then((evt) => rec.pause());
    // expose a Promise for when it's done
    this._init = waitForEvent(rec, 'pause');

  }
  async recordFrame() {

    await this._init; // we have to wait for the recorder to be paused
    const rec = this.recorder;
    const canvas = this.canvas;
    const source = this.source;
    const ctx = this.drawingContext;
    if (canvas.width !== source.width ||
      canvas.height !== source.height) {
      canvas.width = source.width;
      canvas.height = source.height;
    }

    // start our timer now so whatever happens between is not taken in account
    const timer = wait(1000 / this.FPS);

    // wake up the recorder
    rec.resume();
    await waitForEvent(rec, 'resume');

    // draw the current state of source on our internal canvas (triggers requestFrame in Chrome)
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    ctx.drawImage(source, 0, 0);
    // force write the frame
    this.track.requestFrame();

    // wait until our frame-time elapsed
    await timer;

    // sleep recorder
    rec.pause();
    await waitForEvent(rec, 'pause');

  }
  async export () {

    this.recorder.stop();
    this.stream.getTracks().forEach((track) => track.stop());
    await waitForEvent(this.recorder, "stop");
    return new Blob(this.chunks);

  }
}

///////////////////
// how to use:
(async() => {
  const FPS = 30;
  const duration = 5; // seconds

  let x = 0;
  let frame = 0;
  const ctx = canvas.getContext('2d');
  ctx.textAlign = 'right';
  draw(); // we must have drawn on our canvas context before creating the recorder

  const recorder = new FrameByFrameCanvasRecorder(canvas, FPS);

  // draw one frame at a time
  while (frame++ < FPS * duration) {
    await longDraw(); // do the long drawing
    await recorder.recordFrame(); // record at constant FPS
  }
  // now all the frames have been drawn
  const recorded = await recorder.export(); // we can get our final video file
  vid.src = URL.createObjectURL(recorded);
  vid.onloadedmetadata = (evt) => vid.currentTime = 1e100; // workaround https://crbug.com/642012
  download(vid.src, 'movie.webm');

  // Fake long drawing operations that make real-time recording impossible
  function longDraw() {
    x = (x + 1) % canvas.width;
    draw(); // this triggers a bug in Chrome
    return wait(Math.random() * 300)
      .then(draw);
  }

  function draw() {
    ctx.fillStyle = 'white';
    ctx.fillRect(0, 0, canvas.width, canvas.height);
    ctx.fillStyle = 'black';
    ctx.fillRect(x, 0, 50, 50);
    ctx.fillText(frame + " / " + FPS * duration, 290, 140);
  };
})().catch(console.error);

<canvas id="canvas"></canvas>
<video id="vid" controls></video>

<script>
  // Some helpers
  
  // Promise based timer
  function wait(ms) {
    return new Promise(res => setTimeout(res, ms));
  }
  // implements a sub-optimal monkey-patch for requestPostAnimationFrame
  // see https://stackoverflow.com/a/57549862/3702797 for details
  if (!window.requestPostAnimationFrame) {
    window.requestPostAnimationFrame = function monkey(fn) {
      const channel = new MessageChannel();
      channel.port2.onmessage = evt => fn(evt.data);
      requestAnimationFrame((t) => channel.port1.postMessage(t));
    };
  }
  // Promisifies EventTarget.addEventListener
  function waitForEvent(target, type) {
    return new Promise((res) => target.addEventListener(type, res, {
      once: true
    }));
  }
  // creates a downloadable anchor from url
  function download(url, filename = "file.ext") {
    a = document.createElement('a');
    a.textContent = a.download = filename;
    a.href = url;
    document.body.append(a);
    return a;
  }
</script>

这篇关于即使在慢速计算机上也可以使用 CanvasCaptureMediaStream 以恒定 fps 录制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！