Android MediaCodec输出格式:从GLES外部纹理(YUV/NV12)到GLES纹理(RGB) [英] Android MediaCodec output format: GLES External Texture (YUV / NV12) to GLES Texture (RGB)

问题描述

我目前正在尝试在Android上开发视频播放器，但在色彩格式方面苦苦挣扎.

I am currently trying to develop a video player on Android, but am struggling with color formats.

上下文:我通过 MediaExtractor/MediaCodec 的标准组合来提取和解码视频.因为我需要将提取的帧用作OpenGLES纹理(RGB)，所以我设置了解码器( MediaCodec )，以便它可以馈送外部GLES纹理( GL_TEXTURE_EXTERNAL_OES ).我知道硬件解码器输出的数据为 NV12 ( YUV420SemiPlanar )格式，我需要通过渲染将其转换为RGB(使用片段着色器转换).

Context: I extract and decode a video through the standard combinaison of MediaExtractor/MediaCodec. Because I need the extracted frames to be available as OpenGLES textures (RGB), I setup my decoder (MediaCodec) so that it feeds an external GLES texture (GL_TEXTURE_EXTERNAL_OES) through a SurfaceTexture. I know the data output by my HW decoder is in the NV12 (YUV420SemiPlanar) format, and I need to convert it to RGB by rendering it (with a fragment shader doing the conversion).

MediaCodec ---> GLES外部纹理(NV12) [1] --->渲染---> GLES纹理(RGB)

MediaCodec ---> GLES External Texture (NV12) [1] ---> Rendering ---> GLES Texture (RGB)

我遇到的困难:如何访问GLES外部纹理( [1] )中包含的特定Y，U和V值.我不知道如何设置GLES纹理内存，也不知道如何访问它(GLSL函数"texture()"和"texelFetch()"除外).

The point where I struggle is: How do I access the specific Y, U, and V values contained in the GLES External Texture ([1]). I have no idea how the GLES texture memory is set, nor how to access it (except for the "texture()" and "texelFetch()" GLSL functions).

• 有没有一种方法可以像访问简单数组(指针+偏移量)那样访问数据?
• 我是否全盘考虑了?
• Surface 或 SurfaceTexture 是否处理转换? (我不这么认为)
• 在填充GLES外部纹理( [1] )时， Surface 或 SurfaceTexture 更改数据的内存布局，以便组件可以是否可以通过GLES纹理访问功能进行访问?

• Is there a way to access the data as I would access a simple array (pointer + offset)?
• Did I overthink the whole thing?
• Do either Surface or SurfaceTexture take care of conversions? (I don't think so)
• Do either Surface or SurfaceTexture change the memory layout of the data while populating the GLES External Texture ([1]) so components can be accessed through GLES texture access functions?

推荐答案

是的，我想你是在想它.您是否进行了测试并遇到了可以描述的实际问题，或者到目前为止，这仅仅是理论上的吗?

Yes, I would say you're overthinking it. Did you test things and run into an actual issue that you could describe, or is this only theoretical so far?

即使原始解码器本身输出NV12，当您通过SufaceTexture访问它时，此细节也被隐藏-然后您可以将其作为任何RGB纹理访问.由于纹理的物理内存布局是隐藏的，因此您并不真正知道在获取纹理之前是否实际上一次将其全部转换了，或者您每次对其进行采样时纹理访问器是否都在进行即时转换.据我所知，该实现可以通过任何一种方式自由地实现，而且关于实现方式的实现细节根本无法通过公共API来观察.

Even though the raw decoder itself outputs NV12, this detail is hidden when you access it via a SufaceTexture - then you get to access it as any RGB texture. Since the physical memory layout of the texture is hidden, you don't really know if it actually is converted all at once before you get it, or if the texture accessors do a on-the-fly conversion each time you sample it. As far as I know, the implementation is free to do it in any of these ways, and the implementation details about how it is done are not observable through the public APIs at all.

这篇关于Android MediaCodec输出格式:从GLES外部纹理(YUV/NV12)到GLES纹理(RGB)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！