Canvas getImageData在某些移动设备上返回不正确的数据 [英] Canvas getImageData returning incorrect data on certain mobile devices
问题描述
我正在制作一个画布视频播放器，它具有一些基于视频帧的特殊功能。为了克服视频HTML5标签中不可靠的时序，我们使用的视频在每个帧中都嵌入了条形码，表示当前帧编号。使用canvas getImageData方法，我可以抓取像素并读取条形码以获取帧编号。这很好用，我有一个
编辑3：发现
我相信我已经做了一个相关的发现/实现，我之前应该已经看过了。
当看到破碎的getImageData的输出时设备我一步一步地走过去。我没有（也应该）注意到的是，视频元素在达到我的断点/调试器语句之后仍在继续。到执行getImageData方法时，视频已移过下一帧。因此，扫描的条形码实际上是比预期更晚的帧。
我添加了一些控制台日志语句，让它自然运行。看看输出，我可以看到一个更容易识别的模式。
以下是谷歌像素的前几个读数：
Uint8ClampedArray（64）[255,255,255,255,246,246,246,255,243,243,243,255,240,240,241,255,241,241,242,255,241,241,242 ，255,241,241,242,255,241,241,242,255,241,241,242,255,241,241,242,255,241,241,242,255,241,241,242,255 ，241,241,242,255,241,241,242,255,241,241,242,255,2,2,225]
Uint8ClampedArray（64）[5,5， 5,255,255,255,255,255,251,251,251,255,247,247,248,255,245,245,245,255,245,245,245,255,246,246,246， 255,246,246,246,255,246,246,246,255,246,246,246,255,246,246,246,255,246,246,246,255,246,246,246,255， 246,246,246,255,246,246,246,255,6,3,2,255]
Uint8ClampedArray（64）[235,231,230,255,17,12,12] ，255,252,247,247,255,255,255,255,255,255,254,254,255,255,253,253,255,255,252,251,255,255,253,253,255 ，255,253,253,255,255,253,25 3,255,255,253,253,255,255,253,253,255,255,253,253,255,255,253,253,255,255,253,253,255,7,3,1， 255]
Uint8ClampedArray（64）[26,15,14,255,4,0,0,255,255,255,255,255,255,255,255,255,255,255 ，255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255 ，255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,10,0,25]
您可能会注意到，结果似乎是正确的，但它们向左移动了一个像素。
我稍微修改了 JSFiddle 以移动getImageData读取一个像素，它给出与Pixel完全相同的响应。
var pixels = ctx.getImageData（1,320 - 1 ，16,1）.data;
执行-1似乎无效。
因此，由于某种原因，这些设备要么将整个纹理移动一个像素，要么getImageData方法有问题。
编辑4
作为实验，我重新配置了我的代码以使用webGL纹理。桌面/移动设备上的行为相同。这允许我使用gl.readPixels将-1用作x目标。我希望通过跳过使用画布，整个图像将存储在内存中，我可以访问我需要的像素数据....没有工作，但这里是它生成的数据，表明它也使用纯webGL移动。
Uint8Array（64）[0,0,0,0,255,248,248,255,25,18,18 ，255,254,254,254,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255 ，255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255 ，255,255,255]
Uint8Array（64）[0,0,0,0,255,254,247,255,255,244,236,255,48,18,10， 255,254,246,238,255,255,247,239,255,255,247,239,255,255,248,241,255,255,251,243,255,255,251,243,255， 255,251,243,255,255,251,243,255,255,251,243,255,255,251,243,255,255,250,240,255,255,250,240,255]
Uint8Array（64）[0,0,0,0,30,0,0,255,254,243,230,255,43,6,1,255,254,243,230,255 ，255,244,231,255,255,244,231,255,255,244,231,255,255,244,231,255,255,244,231,255,255,244,231,255,255 ，244,231,255,255,244,231,255,255,244,231,255,255,244,229,255,25 5,244,229,255]
使用：
gl.readPixels（-1，height，16,1，gl.RGBA，gl.UNSIGNED_BYTE，pixels）;
编辑5
好的，我保证我会得到更多有关哪些设备出现故障/未出现故障的详细信息。我使用略微修改的 JSFiddle 进行了一些在线QA测试。我对它做了一些修改，以帮助公众更好地证明这一点。
不幸的是，答案相当复杂。我希望它能被隔离到Android 7，但似乎并非如此。
我有 CSV ，带有此测试的结果。并不是说这些测试是100％可靠的，但它似乎只是一些随机设备......
我有一直在经历同样的问题而且无法得到它。在我的案例中，我终于得到了答案，这听起来也像你的问题一样。
这是像素密度!!
所有像素密度都不同你提到的设备和可能有1 dpr（设备像素比）的设备正常工作，而其他设备没有。
所以在我的情况下使用p5js 我将像素密度设置为1，它就像一个魅力;
pixelDensity（1）;
因此将其设置为1 dpr并且很可能很好go！
我希望这可以帮助一些人，因为我花了很长时间来解决这个问题。
I am working on a canvas video player with some special features based on frames of the video. To overcome the unreliable timing in the video HTML5 tag the videos we are using have a barcode embedded in each frame indicating the current frame number. Using the canvas getImageData method I can grab the pixels and read the barcode to get the frame number. This works great and I have a JSFiddle demonstrating that it works (I couldn't get around CORS in this fiddle to serve the video to the canvas so to see it working you'll have to download the example video locally then upload it via the button. Not ideal but it works).
On certain mobile devices (only Android thus far) this logic breaks. The getImageData returns incorrect values.
It works correctly on my Samsung Galaxy S5 v6.0.1 but fails on a Google Pixel running android v7.1.2. I'll try to collect more data on which devices/OS versions it fails on.
For example, when playing on desktop the first iteration of getImageData returns:
Uint8ClampedArray(64) [3, 2, 3, 255, 255, 255, 255, 255, 246, 245, 247, 255, 243, 242, 243, 255, 241, 239, 241, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255]
which correctly gets computed as framenumber 1.
However on the galaxy, the first iteration returns:
Uint8ClampedArray(64) [255, 242, 217, 255, 255, 234, 209, 255, 41, 1, 1, 255, 254, 235, 210, 255, 255, 234, 209, 255, 50, 4, 0, 255, 254, 240, 215, 255, 255, 248, 224, 255, 255, 249, 225, 255, 255, 251, 232, 255, 255, 252, 233, 255, 255, 252, 233, 255, 255, 253, 234, 255, 255, 255, 237, 255, 255, 255, 237, 255, 28, 1, 1, 255]
I read that certain devices may being doing additional smoothing so I've been playing around with disabling it in the context via:
this.ctx.mozImageSmoothingEnabled = false;
this.ctx.webkitImageSmoothingEnabled = false;
this.ctx.msImageSmoothingEnabled = false;
this.ctx.imageSmoothingEnabled = false;
But it didn't help.
Here is the code being used in the JSFiddle.
var frameNumberDiv = document.getElementById('frameNumber');
function load() {
var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
var video = document.getElementById('video');
canvas.width = 568;
canvas.height = 640;
video.addEventListener('play', function() {
var that = this; //cache
(function loop() {
if (!that.paused && !that.ended) {
ctx.drawImage(that, 0, 0);
var pixels = ctx.getImageData(0, 320 - 1, 16, 1).data;
getFrameNumber(pixels);
setTimeout(loop, 1000 / 30); // drawing at 30fps
}
})();
}, 0);
}
function getFrameNumber(pixels) {
let j = 0;
let thisFrameNumber = 0;
let str = "Pixels: ";
for (let i = 0; i < 16; i++) {
str += pixels[j] + " ";
thisFrameNumber += getBinary(pixels[j], i);
j += 4;
}
document.getElementById('frameNumber').innerHTML = "FrameNumber: " + thisFrameNumber;
}
function getBinary(pixel, binaryPlace) {
const binary = [1, 2, 4, 8, 16, 32, 64, 128, 256,
512, 1024, 2048, 4096, 8192, 16384, 32768
];
if (pixel > 128) return 0;
if (pixel < 128 && binary[binaryPlace]) {
return binary[binaryPlace]
} else {
return 0;
}
}
(function localFileVideoPlayer() {
'use strict';
var URL = window.URL || window.webkitURL;
var displayMessage = function(message, isError) {
var element = document.querySelector('#message');
element.innerHTML = message;
element.className = isError ? 'error' : 'info';
}
var playSelectedFile = function(event) {
console.log("Playing");
var file = this.files[0];
var type = file.type;
var videoNode = document.querySelector('video');
var canPlay = videoNode.canPlayType(type);
if (canPlay === '') canPlay = 'no';
var message = 'Can play type "' + type + '": ' + canPlay;
var isError = canPlay === 'no';
displayMessage(message, isError);
if (isError) {
return;
}
var fileURL = URL.createObjectURL(file);
videoNode.src = fileURL;
load();
}
var inputNode = document.querySelector('input')
inputNode.addEventListener('change', playSelectedFile, false)
})();
EDIT
- Works on a Nexus 6P running Android v6.0
- Works on a Samsung 6 (Samsung SM G920A) running Android v 5.0.2
- It DOES NOT work on a Samsung Galaxy S7 (SAMSUNG-SM-G935A) running Android v7.0
Could this possibly be an Android 7 issue?
Edit 2
In response to a question in the comments:
videoNode.videoHeight and videoWidth are both 0 on the google pixel for their entire existence but this is the same as on desktop. In both of the devices that don't work that I've encountered the image of each frame is fully painted. I'll attach a screen shot from the google pixel. When paused it consistently reads the same number. In other words it is not jumping around so whatever it is reading is truly on the frame of the video.
EDIT 3: Discovery
I believe that I've made a relevant discovery/realization which I should have seen earlier.
When looking at the output of getImageData on the broken device I was stepping through line by line. What I hadn't (and should have) noticed was that the video element was continuing slightly after it hit my break points / debugger statements. By the time the getImageData method was executed the video had moved past the next frame. So, the scanned barcode was actually for a much later frame than expected.
I added some console log statements and let it run naturally. Looking at the output I can see a much more recognizable pattern.
Here is the first few readings on the google pixel:
Uint8ClampedArray(64) [255, 255, 255, 255, 246, 246, 246, 255, 243, 243, 243, 255, 240, 240, 241, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 2, 2, 2, 255]
Uint8ClampedArray(64) [5, 5, 5, 255, 255, 255, 255, 255, 251, 251, 251, 255, 247, 247, 248, 255, 245, 245, 245, 255, 245, 245, 245, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 6, 3, 2, 255]
Uint8ClampedArray(64) [235, 231, 230, 255, 17, 12, 12, 255, 252, 247, 247, 255, 255, 255, 255, 255, 255, 254, 254, 255, 255, 253, 253, 255, 255, 252, 251, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 7, 3, 1, 255]
Uint8ClampedArray(64) [26, 15, 14, 255, 4, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 10, 0, 0, 255]
As you may notice, the results seem to be correct however they are shifted one pixel to the left.
I modified the JSFiddle slightly to shift the getImageData read over by a pixel and it gives the exact same response as on the Pixel.
var pixels = ctx.getImageData(1, 320 - 1, 16, 1).data;
Doing -1 seems to have no effect.
So, for some reason these devices are either shifting the entire texture over by a pixel or there is something wrong with the getImageData Method.
EDIT 4
As an experiment I reconfigured my code to use a webGL texture. Same behaviour on desktop/mobile devices. This allowed me to use -1 as the x target using gl.readPixels. I was hoping that by skipping using canvas the entire image would be stored in memory and I could access the pixel data I needed.... Didn't work but here is the data it produced which shows that it is also shifted using purely webGL.
Uint8Array(64) [0, 0, 0, 0, 255, 248, 248, 255, 25, 18, 18, 255, 254, 254, 254, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255]
Uint8Array(64) [0, 0, 0, 0, 255, 254, 247, 255, 255, 244, 236, 255, 48, 18, 10, 255, 254, 246, 238, 255, 255, 247, 239, 255, 255, 247, 239, 255, 255, 248, 241, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 250, 240, 255, 255, 250, 240, 255]
Uint8Array(64) [0, 0, 0, 0, 31, 0, 0, 255, 254, 243, 230, 255, 43, 6, 1, 255, 254, 243, 230, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 229, 255, 255, 244, 229, 255]
Using:
gl.readPixels(-1, height, 16, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
Edit 5
Ok, I promised I would get more details on which devices were failing/not. I had some online QA testing done using a slightly modified JSFiddle. I modified it slightly to help make it a bit more idiot proof for the general public to work with.
The responses were unfortunately fairly mixed. I was hoping it would be isolated to Android 7 but that doesn't seem to be the case.
I have a CSV on my google drive with the results of this test. Not that these tests are 100% reliable but it seems like it's just some random devices......
I have been going through the same problem and just couldn't get it. I finally have the answer in my case which sounds 99% like your problem as well.
It's the pixel density!!
The pixel densities are different on all the devices you mentioned and the ones that have probably 1 dpr (device pixel ratio) are working correctly, whereas the others are not.
So in my case using p5js I set the pixel density to be 1 and it worked like a charm;
pixelDensity(1);
So set it to be 1 dpr and you are most probably good to go!
I hope this helps some people out there because I spent quite a while on this problem.
这篇关于Canvas getImageData在某些移动设备上返回不正确的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
