Chrome扩展程序captureStream [英] Chrome Extension captureStream

问题描述

我正在开发chrome扩展，其中包含以下内容：

$ ul

访问从<$ c $获得的流c> chrome.tabCapture.capture （捕捉视频时，让我们暂时忽略音频捕捉，因为它与我所面临的问题无关）

li>

传递 tabStream 到 URL.createObjectURL（tabStream）

为DOM视频元素 videoEl.src = URL.createObjectURL（tabStream）使用生成的url作为 src

invoke videoEl.play（） c> canplay 事件被称为

传递 videoEl 作为参数到canvas的上下文 drawImage method

现在视频帧被渲染到canvas元素中，对这些帧进行有用的操作（裁剪，水印等）。

b
$ b

直到这一点完全奏效。但是以下两个最后步骤不起作用：
$ b $ ul

使用创建一个流出canvas元素， canvasStream = canvasEl.captureStream（20）

将此流传递给 MediaRecorder （ recorder = new MediaRecorder（canvasStream））并开始录制： recorder.start（）

基本上，如果这种方法在Chrome扩展的背景之外使用（例如： https://jsfiddle.net/williamwallacebrave/2rgv7pgj/7/ ）完美无缺。但是，当在Chrome扩展背景中使用时，我可以清楚地检测到视频帧是在canvas元素中发送和渲染的，但不管怎样， canvasEl.captureStream（）数据或记录器无法提取它们。此外，如果在内容脚本中再次使用该方法，则所有方法都可以完美地工作但在内容脚本中，我无法访问 tabCapture 流。

这是我的清单文件：

  {
name：super app，
manifest_version：2，
description：...，
version：0.0.1，
content_security_policy：script-src'self''unsafe-eval'; object-src 'self'，
page_action：{
default_title：app，
default_icon：static / images / logo.png
}，
图标：{
128：static / images / logo.png
}，
background：{
page：background .html
，
content_scripts：[
 {
matches：[< all_urls>]，
exclude_matches：[ http：// localhost：3000 / *]，
css：[
static / css / style.css
]，
js：[
vendor / system.js，
 vendor / jquery.min.js，
content / config.js，
content / index.js
] 
} 
]，
web_accessible_resources：[
background / *，
vendor / *，
content / *，
common / *。js， 
capturer.html，
static / *，
*
]，
externally_connectable：{
matches： [
http：// localhost：3000 / *
] 
}，
permissions：[
tabs，
activeTab ，
all_urls>，
clipboardRead，
clipboardWrite，
tabCapture，
notifications，
 tts

} 
  

这是伪代码，内容脚本工作得很好，但不能用作背景：

  //源代码：http://stackoverflow.com/问题/ 39302814 / mediastream-capture-canvas-and-audio-simultaneous＃39302994 
 var cStream，
 aStream，
记录器，
 chunks = []，
 canvasEl =文档.createElement（ '画布'）; 
 
 canvasEl.width = 400; 
 canvasEl.height = 400; 
 document.body.appendChild（canvasEl）; 
 $ b $ * / 
创建并运行外部视频
 * / 
 var videoEl = document.createElement（'video'）; 
 videoEl.crossOrigin ='anonymous'; 
 videoEl.src ='https://dl.dropboxusercontent.com/s/bch2j17v6ny4ako/movie720p.mp4'; 
 videoEl.play（）; 
 videoEl.addEventListener（'play'，function（）{
 var audioCtx = new AudioContext（）; 
 var dest = audioCtx.createMediaStreamDestination（）; 
 
 aStream = dest.stream; 
 var sourceNode = audioCtx.createMediaElementSource（this）; 
 console.log（'connected audio'）; 
 sourceNode.connect（dest）; 
 
 //输出到我们的耳机
 sourceNode.connect（audioCtx.destination）
 
 var canvasCtx = canvasEl.getContext（'2d'）; 
 console.log（'play （），
 draw（this，canvasCtx）; 
 
 startRecording（）; 
 setTimeout（（）=> {
 stopRecording（）; 
}，10000）
 
}，false）; 
 
 
函数exportStream（e）{
 console.log（'exportStream'，chunks.length）; 
 if（chunks.length）{
 var blob = new Blob（chunks），
 videoURL = URL.createObjectURL（blob），
 resultVideoEl = document.createElement（'video' ）; 
 
 resultVideoEl.controls = true; 
 resultVideoEl.src = videoURL; 
 resultVideoEl.onend = function（）{
 URL.revokeObjectURL（videoURL）; 
} 
 document.body.insertBefore（resultVideoEl，canvasEl）; 
 
} else {
 document.body.insertBefore（
 document.createTextNode（'no data saved'），canvasEl）; 
 
 
 
 $ b函数saveChunks（e）{
 console.log（'save chunks'，e.data.size）; 
 e.data.size&& chunks.push（e.data）; 
 
 
 
 function stopRecording（）{
 console.log（'STOP RECORDING'）; 
 videoEl.pause（）; 
 recorder.stop（）; 
} 
 
 
函数startRecording（）{
 console.log（'START RECORDING'）; 
 cStream = canvasEl.captureStream（30）; 
 cStream.addTrack（aStream.getAudioTracks（）[0]）; 
 
 recorder = new MediaRecorder（cStream）; 
 recorder.start（）; 
 
 // ======================================== ===== 
 //这部分不会在后台运行时被烧制
 //最终块总是一个空数组。 
 // ============================================ = 
 recorder.ondataavailable = saveChunks; 
 recorder.onstop = exportStream; 
 
 
 
函数draw（v，ctx）{
 if（videoEl.paused || videoEl.ended）return false; 
 
 //这里我裁剪视频帧，并且只需要400到400美元b $ b //平方移动100,100向量
 ctx.drawImage（v，100，100 ，400，400，0，0，400，400）; 
 setTimeout（draw，20，v，ctx）; 
} 
  

另外请注意，CaptureStream和MediaRecorder相对较新，所以您需要Chrome 51 +为了运行该示例

解决方案

这很可能与我在几个月前提交的这个Chromium错误有关： https://bugs.chromium.org/p/chromium/issues/detail？ ID = 639105 。

以下是负责此API的Chrome工程师的回复：

如果Canvas在背景渲染时不绘制/渲染新框架，并且作为
结果，则画布捕获不包含任何新项目。 AFAIK，
，现在无法像使用Chrome一样对代码进行背景渲染
的绘制和捕获画布。

请将您的使用案例发布到那里，并显示您对此问题的支持。

I'm developing chrome extension which makes the following:

• get access to the stream obtained from chrome.tabCapture.capture (when capturing video, let's ignore for now audio capture since it's not relevant to the issue I'm facing)

• pass this tabStream to URL.createObjectURL(tabStream)

• use the resulting url as a src for DOM Video Element videoEl.src = URL.createObjectURL(tabStream)

• invoke videoEl.play() and when the canplay event is called

• pass the videoEl as an argument to canvas's context drawImage method

• since now video frames are rendered into the canvas element one can perform lots of useful operations on that frames (cropping, watermarking etc)

till this point all works perfectly. But the following two final steps do not work:

• create a stream out of the canvas element using canvasStream = canvasEl.captureStream(20)

• pass this stream to MediaRecorder (recorder = new MediaRecorder(canvasStream)) and start recording: recorder.start()

Essentially if this approach is used outside of chrome extension's background (like here: https://jsfiddle.net/williamwallacebrave/2rgv7pgj/7/) all works perfectly. But when used inside the chrome extension background, well I can clearly detect that video frames are send and rendered in canvas element but somehow either the canvasEl.captureStream() is not pushing the data or recorder is not able to pick them up. Also if that approach is used inside the content scripts again all works perfectly. But in content scripts I'm not able to get access to tabCapture stream.

This is my manifest file:

{
    "name": "super app",
    "manifest_version": 2,
    "description": "...",
    "version": "0.0.1",
    "content_security_policy": "script-src 'self' 'unsafe-eval'; object-src 'self'",
    "page_action": {
        "default_title": "app",
        "default_icon": "static/images/logo.png"
    },
    "icons": {
        "128": "static/images/logo.png"
    },
    "background": {
        "page": "background.html"
    },
    "content_scripts": [
        {
            "matches": ["<all_urls>"],
            "exclude_matches": ["http://localhost:3000/*"],
            "css": [
                "static/css/style.css"
            ],
            "js": [
                "vendor/system.js",
                "vendor/jquery.min.js",
                "content/config.js",
                "content/index.js"
            ]
        }
    ],
    "web_accessible_resources": [
        "background/*",
        "vendor/*",
        "content/*",
        "common/*.js",
        "capturer.html",
        "static/*",
        "*"
    ],
    "externally_connectable": {
        "matches": [
            "http://localhost:3000/*"
        ]
    },
    "permissions": [
        "tabs",
        "activeTab",
        "<all_urls>",
        "clipboardRead",
        "clipboardWrite",
        "tabCapture",
        "notifications",
        "tts"
    ]
}

Here is the dummy code which when run as a content script works perfectly fine but doesn't work as background:

// SOURCE: http://stackoverflow.com/questions/39302814/mediastream-capture-canvas-and-audio-simultaneously#39302994
var cStream,
    aStream,
    recorder,
    chunks = [],
    canvasEl = document.createElement('canvas');

canvasEl.width = 400;
canvasEl.height = 400;
document.body.appendChild(canvasEl);

/*
   create and run external video
*/
var videoEl = document.createElement('video');
videoEl.crossOrigin = 'anonymous';
videoEl.src = 'https://dl.dropboxusercontent.com/s/bch2j17v6ny4ako/movie720p.mp4';
videoEl.play();
videoEl.addEventListener('play', function(){
    var audioCtx = new AudioContext();
    var dest = audioCtx.createMediaStreamDestination();

    aStream = dest.stream;
    var sourceNode = audioCtx.createMediaElementSource(this);
    console.log('connected audio');
    sourceNode.connect(dest);

    // output to our headphones  
    sourceNode.connect(audioCtx.destination)

    var canvasCtx = canvasEl.getContext('2d');
    console.log('play video in canvas');
    draw(this, canvasCtx);

    startRecording();
    setTimeout(() => {
        stopRecording();
    }, 10000)      

}, false);


function exportStream(e) {
    console.log('exportStream', chunks.length);
    if (chunks.length) {
        var blob = new Blob(chunks),
            videoURL = URL.createObjectURL(blob),
            resultVideoEl = document.createElement('video');

        resultVideoEl.controls = true;
        resultVideoEl.src = videoURL;
        resultVideoEl.onend = function() {
            URL.revokeObjectURL(videoURL);
        }
        document.body.insertBefore(resultVideoEl, canvasEl);

    } else {
        document.body.insertBefore(
            document.createTextNode('no data saved'), canvasEl);
    }
}


function saveChunks(e) {
    console.log('save chunks', e.data.size);
    e.data.size && chunks.push(e.data);
}


function stopRecording() {
    console.log('STOP RECORDING');
    videoEl.pause();
    recorder.stop();
}


function startRecording() {
    console.log('START RECORDING');
    cStream = canvasEl.captureStream(30);
    cStream.addTrack(aStream.getAudioTracks()[0]);

    recorder = new MediaRecorder(cStream);
    recorder.start();

    // =============================================
    // THIS PART IS NOT FIRED WHEN RUN IN BACKGROUND
    // and final chunks is always an empty array. 
    // =============================================
    recorder.ondataavailable = saveChunks;
    recorder.onstop = exportStream;
}


function draw(v,ctx) {
    if(videoEl.paused || videoEl.ended) return false;

    // here I'm cropping the video frames and taking only 400 by 400
    // square shifted by 100, 100 vector
    ctx.drawImage(v, 100, 100, 400, 400, 0, 0, 400,400);
    setTimeout(draw,20,v,ctx);
}

Also please notice that this captureStream and MediaRecorder are relatively new so you need Chrome 51+ in order to run that example

解决方案

This is most likely related to this Chromium bug I submitted a few months ago: https://bugs.chromium.org/p/chromium/issues/detail?id=639105.

Here's the response from the Chrome engineer in charge of this API:

Canvas does not paint/render new frames when backgrounded and as a result, the canvas capture does not contain any new items. AFAIK, there isn't a way to paint&capture canvas when tab is backgrounded right now in the way you expect on the demos using Chrome.

Please post your use case there and show your support for getting this fixed.

这篇关于Chrome扩展程序captureStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！