使用高分辨率本地视频但限制 WebRTC 连接中的视频大小 [英] Use high resolution local video but limit video size in WebRTC connection

问题描述

我正在开发一个网站，用于在移动浏览器上进行一对一的 webrtc 视频通话.我还支持在本地流上拍摄照片.对于照片捕获，我在可用的情况下使用 ImageCapture api，并在其他地方绘制到画布.

I'm developing a website for a one on one webrtc video call on mobile browsers. I also support the capturing of photos on the local stream. For the photo capture I use ImageCapture api where available, and drawing to canvas elsewhere.

问题是我希望视频的分辨率为 1280x720(以减少通话所需的带宽)，而我需要以 1920x1080 的分辨率拍摄照片.

The problem is that I want the video to be HD resolution 1280x720 (to reduce the bandwith needed for the call) while I need to capture the photos at 1920x1080.

我现在要做的是在拍摄照片之前应用约束并将分辨率设置为全高清，拍摄后将分辨率设置回高清.有时发生的情况是因为分辨率的变化，照片没有对焦，因为相机必须重新对焦.

What I do now is that before I capture the photo I apply constraints and set the resolution to Full HD and after capturing it I set the resolution back to HD. What happens sometimes is because of the change in resolution, the photos are not focused since the camera has to refocus.

所以如果我想要更高分辨率的照片，我会看到 2 个选项，其中我已经尝试了第一个:

So if I want higher resolution photos I see 2 options, of which I already tried the first one:

1. 我尝试使用 2 个单独的 MediaStreamTracks，一个分辨率为 1280x720，我通过 WebRTC 连接发送，另一个分辨率为 1920x1080，我用来在本地显示它并捕获照片.这在大多数手机上运行良好，但有些手机的一个轨道有视频，但第二个轨道根本不起作用(它没有显示任何视频)(特别是 iPhone 6s)
2. 如果可能，我只会使用一个分辨率为 1920x1080 的 MediaStreamTrack，并且我会限制连接本身的视频大小，以便它通过 WebRTC 连接发送较低的分辨率.

1. I tried having 2 separate MediaStreamTracks, one with resolution of 1280x720 that I was sending through the WebRTC connection, and one with resolution of 1920x1080 that I used to display it locally and capture the photos. This worked well on most phones, but there were some phones where one track had the video but the second track didn't work at all (it didn't display any video) (specifically iPhone 6s)
2. If possible I would only use one MediaStreamTrack with the resolution of 1920x1080 and I would limit the video size within the connection itself so it would send the lower resolution through the WebRTC connection.

所以我的问题是，是否可以在本地使用更高分辨率的视频，然后限制通过 WebRTC 连接发送的视频大小以减少带宽使用?

So my question is, is it possible to locally use higher resolution video and then limit the video size sent through WebRTC connection to reduce bandwith usage?

推荐答案

是否可以在本地使用更高分辨率的视频，然后限制通过 WebRTC 连接发送的视频大小以减少带宽使用?

is it possible to locally use higher resolution video and then limit the video size sent through WebRTC connection to reduce bandwith usage?

是的，使用 scaleResolutionDownBy，一个参数RTCRtpSender 就是这样做的(iOS 除外)

Yes, use scaleResolutionDownBy, a parameter of RTCRtpSender that does just this (except in iOS)

该参数最初针对多层(联播)传输，在单个编码上也能正常工作.这是一个比率，您可以使用 setParameters 进行设置.还有一个 maxBitrate 您可以直接控制:

This parameter, originally aimed at multi-layer (simulcast) transmissions, works just fine on a single encoding as well. It's a ratio, and you set it with setParameters. There's also a maxBitrate you can control directly:

const sender = pc.addTrack(cameraTrack);

async function setParams(height, bitrate) {
  const ratio = sender.track.getSettings().height / height;

  const params = sender.getParameters();
  params.encodings[0].scaleResolutionDownBy = Math.max(ratio, 1);
  params.encodings[0].maxBitrate = bitrate;
  await sender.setParameters(params);
}

请参阅我的博客以获取工作演示.

See my blog for a working demo.

不幸的是，浏览器围绕此功能存在各种错误和不同阶段的支持:

Unfortunately, browsers have various bugs and different stages of support around this feature:

1. Safari 实现了 maxBitrate 而不是 scaleResolutionDownBy，它总是显示为 1.
2. Firefox 支持两者，但仍使用需要一些调整的早期版本的 setParameters.

1. Safari implements maxBitrate but not scaleResolutionDownBy which always shows as 1.
2. Firefox supports both, but still uses an early version of setParameters that needs some tweaks.

我们通过为 Firefox 添加一个小的调整来解决这个问题，并回退到使用 track.clone 和 applyConstraints 在 Safari 中缩小规模:

We work around this by adding a minor tweak for Firefox, and by falling back to using track.clone and applyConstraints to downscale in Safari:

// With workarounds for Firefox and Safari

const sender = pc.addTrack(cameraTrack.clone()); // Note we clone the track

async function setParams(height, bitrate) {
  const ratio = sender.track.getSettings().height / height;

  const params = sender.getParameters();
  if (!params.encodings) {
    params.encodings = [{}]; // Firefox workaround
  }
  params.encodings[0].scaleResolutionDownBy = Math.max(ratio, 1);
  params.encodings[0].maxBitrate = bitrate;
  await sender.setParameters(params);

  // Safari fallback
  if (sender.getParameters().encodings[0].scaleResolutionDownBy == 1) {
    await sender.track.applyConstraints({height});
  }
}

这是早期演示的调整版，应该适用于所有浏览器，包括 Safari在 macOS 上.

Here's a tweaked version of the earlier demo that should work in all browsers, including Safari on macOS.

注意:一旦 Safari 添加了对 scaleResolutionDownBy 的原生支持，fiddle 应该会开始使用新功能.在那之前，您很遗憾会遇到与现在相同的 iOS 限制.

Note: Once Safari adds native support for scaleResolutionDownBy the fiddle should pick up on the new functionality. Until then you'll unfortunately run into the same limitation on iOS you're experiencing now.

注意:Safari 14 支持 scaleResolutionDownBy 本机(但不确定 iOS).

Note: Safari 14 supports scaleResolutionDownBy natively (but unsure about iOS).

这篇关于使用高分辨率本地视频但限制 WebRTC 连接中的视频大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！