Webgl 在没有绘制调用的情况下将纹理数据上传到 GPU [英] Webgl Upload Texture Data to the gpu without a draw call
问题描述
我正在使用 webgl 在自定义视频编解码器上进行 YUV 到 RGB 的转换.
I'm using webgl to do YUV to RGB conversions on a custom video codec.
视频必须以 30 fps 播放.为了实现这一点，我每隔一个 requestAnimationFrame 就做我所有的数学计算.
The video has to play at 30 fps. In order to make this happen I'm doing all my math every other requestAnimationFrame.
这很好用，但我注意到在分析时将纹理上传到 gpu 花费的时间最长.
This works great, but I noticed when profiling that uploading the textures to the gpu takes the longest amount of time.
所以我分别上传了Y"纹理和UV"纹理.
So I uploaded the "Y" texture and the "UV" texture separately.
现在第一个requestAnimationFrame"将像这样上传Y"纹理:
Now the first "requestAnimationFrame" will upload the "Y" texture like this:
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, yTextureRef);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, textureWidth, textureHeight, 0, gl.LUMINANCE, gl.UNSIGNED_BYTE, yData);
第二个requestAnimationFrame"将以同样的方式上传UV"纹理，并对片段着色器进行绘制调用，在它们之间进行数学运算.
The second "requestAnimationFrame" will upload the "UV" texture in the same way, and make a draw call to the fragment shader doing the math between them.
但这不会改变分析器中的任何内容.我仍然在上传Y"纹理的帧上显示近 0 gpu 时间，并且在上传UV"纹理的帧上显示的时间与以前相同.
But this doesn't change anything in the profiler. I still show nearly 0 gpu time on the frame that uploads the "Y" texture, and the same amount of time as before on the frame that uploads the "UV" texture.
但是，如果我向Y"纹理上传函数添加绘制调用，则分析器会显示预期结果.每一帧都有将近一半的 GPU 时间.
However if I add a draw call to my "Y" texture upload function, then the profiler shows the expected results. Every frame has nearly half the gpu time.
由此我猜测 Y 纹理并没有真正使用 texImage2d 函数上传到 GPU.
From this I'm guessing the Y texture isn't really uploaded to the gpu using the texImage2d function.
然而，我真的不想在屏幕上绘制 Y 纹理，因为它没有正确的 UV 纹理，直到一帧之后才能做任何事情.那么有没有办法强制gpu上传这个纹理而不执行draw call?
However I don't really want to draw the Y texture on the screen as it doesn't have the correct UV texture to do anything with until a frame later. So is there any way to force the gpu to upload this texture without performing a draw call?
推荐答案
更新
我误解了这个问题
Update
I mis-understood the question
这真的取决于驱动程序.问题是 OpenGL/OpenGL ES/WebGL 的纹理 API 真的很烂.Sucks 是一个技术术语，表示有意想不到的后果".
It really depends on the driver. The problem is OpenGL/OpenGL ES/WebGL's texture API really sucks. Sucks is a technical term for 'has unintended consequences'.
问题是驱动程序在您绘制之前无法真正完整地上传数据，因为它不知道您将要更改哪些内容.您可以以任何顺序和任何大小更改所有 mip 级别，然后在两者之间修复它们，因此直到您绘制它为止，您都不知道您将调用哪些其他函数来操作纹理.
The issue is the driver can't really fully upload the data until you draw because it doesn't know what things you're going to change. You could change all the mip levels in any order and any size and then fix them all in between and so until you draw it has no idea which other functions you're going to call to manipulate the texture.
考虑您创建一个 4x4 级别 0 mip
Consider you create a 4x4 level 0 mip
gl.texImage2D(
gl.TEXTURE_2D,
0, // mip level
gl.RGBA,
4, // width
4, // height
...);
它应该分配什么内存?4(宽)* 4(高)* 4(RGBA)?但是如果你调用 gl.generateMipmap 呢?现在它需要 4*4*4+2*2*4+1*1*4.好的，但现在您在第 3 级分配了一个 8x8 mip.您打算然后分别用 64x64、32x32、16x16 替换第 0 到 2 级，但您先做了第 3 级.在替换上面的级别之前替换级别 3 应该怎么做?然后添加级别 4 8x8、5 为 4x4、6 为 2x2、7 为 1x1.
What memory should it allocate? 4(width) * 4(height) * 4(rgba)? But what if you call gl.generateMipmap? Now it needs 4*4*4+2*2*4+1*1*4. Ok but now you allocate an 8x8 mip on level 3. You intend to then replace levels 0 to 2 with 64x64, 32x32, 16x16 respectively but you did level 3 first. What should it do when you replace level 3 before replacing the levels above those? You then add in levels 4 8x8, 5 as 4x4, 6 as 2x2, and 7 as 1x1.
如您所见，API 允许您以任何顺序更改 mips.事实上，我可以将级别 7 分配为 723x234，然后再修复它.该 API 旨在直到绘制时间才关心所有 mips 必须具有正确的大小，此时它们可以最终在 GPU 上分配内存并复制 mips.
As you can see the API lets you change mips in any order. In fact I could allocate level 7 as 723x234 and then fix it later. The API is designed to not care until draw time when all the mips must be the correct size at which point they can finally allocate memory on the GPU and copy the mips in.
您可以查看此问题的演示和测试 此处.该测试不按顺序上传 mips，以验证 WebGL 实现是否正确失败，因为它们的大小不正确，并且一旦大小正确就可以正确开始工作.
You can see a demonstration and test of this issue here. The test uploads mips out of order to verify that WebGL implementations correctly fail with they are not all the correct size and correctly start working once they are the correct sizes.
你可以看出这可以说是一个糟糕的 API 设计.
You can see this was arguably a bad API design.
他们添加了 gl.texStorage2D 来修复它，但 gl.texStorage2D 在 WebGL1 中不可用，只有 WebGL2.gl.texStorage2D 有新问题:(
They added gl.texStorage2D to fix it but gl.texStorage2D is not available in WebGL1 only WebGL2. gl.texStorage2D has new issues though :(
TLDR；当您调用 gl.texImage2D 时，纹理会上传到驱动程序，但驱动程序在绘制时间之前无法上传到 GPU.
TLDR; textures get uploaded to the driver when you call gl.texImage2D but the driver can't upload to the GPU until draw time.
可能的解决方案:使用 gl.texSubImage2D 因为它不分配内存，所以驱动程序可以更快地上传.我怀疑大多数驱动程序不会，因为您可以在绘制之前使用 gl.texSubImage2D.还是值得一试
Possible solution: use gl.texSubImage2D since it does not allocate memory it's possible the driver could upload sooner. I suspect most drivers don't because you can use gl.texSubImage2D before drawing. Still it's worth a try
我还要补充一点，gl.LUMIANCE 也可能是一个瓶颈.IIRC DirectX 没有相应的格式，OpenGL Core Profile 也没有.两者都支持 RED only 格式，但 WebGL1 不支持.因此必须通过扩展上传数据来模拟 LUMIANCE.
Let me also add that gl.LUMIANCE might be a bottleneck as well. IIRC DirectX doesn't have a corresponding format and neither does OpenGL Core Profile. Both support a RED only format but WebGL1 does not. So LUMIANCE has to be emulated by expanding the data on upload.
不幸的是，除了通过 texImage2D 和 texSubImage2D
Unfortunately there is no way to upload video to WebGL except via texImage2D and texSubImage2D
有些浏览器会尝试加快速度.我注意到您正在使用 gl.LUMINANCE.您可以尝试使用 gl.RGB 或 gl.RGBA 并查看速度是否加快.浏览器可能只针对更常见的情况进行优化.另一方面，它们可能根本没有优化.
Some browsers try to make that happen faster. I notice you're using gl.LUMINANCE. You might try using gl.RGB or gl.RGBA and see if things speed up. It's possible browsers only optimize for the more common case. On the other hand it's possible they don't optimize at all.
已经提出了两个允许在没有副本的情况下使用视频的扩展，但 AFAIK 没有浏览器实现它们.
Two extensions what would allow using video without a copy have been proposed but AFAIK no browser as ever implemented them.
这实际上比听起来要困难得多.
It's actually a much harder problem than it sounds like.
视频数据可以采用多种格式.你提到了 YUV，但还有其他的.浏览器应该告诉应用程序格式还是浏览器应该转换为标准格式?
Video data can be in various formats. You mentioned YUV but there are others. Should the browser tell the app the format or should the browser convert to a standard format?
告诉的问题是很多开发者会弄错，然后用户会提供他们不支持的格式的视频
The problem with telling is lots of devs will get it wrong then a user will provide a video that is in a format they don't support
WEBGL_video_texture 扩展通过重写您的着色器转换为标准格式.你告诉它 uniform samplerVideoWEBGL video 然后它知道它可以将你的 color = texture2D(video, uv) 重写为 color = convertFromVideoFormatToRGB(texture(video,uv)).这也意味着，如果您播放不同格式的视频，他们就必须即时重写着色器.
The WEBGL_video_texture extensions converts to a standard format by re-writing your shaders. You tell it uniform samplerVideoWEBGL video and then it knows it can re-write your color = texture2D(video, uv) to color = convertFromVideoFormatToRGB(texture(video, uv)). It also means they'd have to re-write shaders on the fly if you play different format videos.
同步
将视频数据传输到 WebGL 听起来很棒，但现在您遇到的问题是，当您获得数据并将其渲染到屏幕时，您已经添加了几帧延迟，因此音频不再同步.
It sounds great to get the video data to WebGL but now you have the issue that by the time you get the data and render it to the screen you've added a few frames of latency so the audio is no longer in sync.
如何处理这超出了 WebGL 的范围，因为 WebGL 与音频没有任何关系，但它确实指出，这并不像仅向 WebGL 提供数据那么简单.一旦您使数据可用，人们就会要求更多 API 来获取音频和更多信息，以便他们可以延迟其中一项或两项并保持同步.
How to deal with that is out of the scope of WebGL as WebGL doesn't have anything to do with audio but it does point out that it's not as simple as just giving WebGL the data. Once you make the data available then people will ask for more APIs to get the audio and more info so they can delay one or both and keep them in sync.
TLDR；除了通过 texImage2D 和 texSubImage2D
这篇关于Webgl 在没有绘制调用的情况下将纹理数据上传到 GPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
