从 HTML5 视频中获取原始像素数据 [英] Get raw pixel data from HTML5 video

问题描述

我想从视频元素中获取原始数据(TypedArray 之类的)并用 JavaScript 操作它们.

I would like to get the raw data (TypedArray or something) from video element and manipulate them with JavaScript.

目前我创建了一个新画布，将视频绘制到画布中，然后获取图像数据.

Currently I create a new canvas, draw the video into canvas and then get the image data.

ctx.drawImage(myVideo);
var data = ctx.getImageData(0, 0, w, h).data;

它工作正常，但会消耗 CPU(将视频放到画布上并从画布复制回来)，并且会产生大量垃圾(每秒大约 50 MB).有没有其他更简单的解决方案?如果我可以将自己的缓冲区传递给 getImageData(...)，那就太好了.

It works fine, but it drains a CPU (putting the video to canvas and copying back from canvas) and it creates a lot of garbage (about 50 MB each second). Is there any other simpler solution? It would be great if I could pass my own buffer to getImageData(...).

顺便说一句.使用 WebGL 绘制视频并从 GPU 加载它并没有更快:( http://jsperf.com/getting-raw-data-from-video

BTW. drawing video with WebGL and loading it back from GPU is not any faster :( http://jsperf.com/getting-raw-data-from-video

推荐答案

请阅读

https://www.scirra.com/blog/76/how-to-write-low-garbage-real-time-javascript

因为您实际上没有显示任何代码、提及浏览器或提供有关视频的信息(分辨率、FPS、编码)，所以无法说明为什么您的代码很慢或为什么它创建显示很多垃圾.使用 Javasrcipt 的实时视频效果在某些分辨率限制下是可能的.

Because you don't actually show any code, mention the browser or give out information about the video (resolution, FPS, encoding) it is impossible to tell why your code is slow or why it creating show much garbage. Real-time video effects with Javasrcipt are possible with some resolution constrain.

这是一个使用 . 的实时视频过滤的 Mozilla 示例.

Here is an Mozilla example of real-time video filtering using .

https://developer.mozilla.org/samples/video/chroma-key/index.xhtml

如果不先在 上传输视频帧，您将无法获得对视频数据的任何原始访问权限.但是，我相信这个操作应该是硬件加速的，因为它发生在 GPU 内存中.从 GPU 下载像素，在 Javascript 中操作它们，然后上传回显示内存可能是高分辨率的最慢步骤.

You won't get any raw access to video data without blitting a video frame on <canvas> first. However, I believe this operation should be HW accelerated as it happens all in GPU memory. Downloading pixels down from GPU manipulating them in Javascript and then uploading back to the display memory might be the slowest step for high resolution.

优化提示

• 考虑跳帧(较低 FPS 中的处理效果)

• Consider skipping frames (process effects in lower FPS)

服务器端 HTML5 视频渲染 服务器端渲染 HTML5 动画?

Server-side HTML5 video rendering Rendering HTML5 animation server-side?

这篇关于从 HTML5 视频中获取原始像素数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！