如何从 MP4 中逐帧获取?(媒体编解码器) [英] How to get frame by frame from MP4? (MediaCodec)

问题描述

实际上我正在使用 OpenGL，我想将我所有的纹理都放在 MP4 中以便压缩它们.

Actually I am working with OpenGL and I would like to put all my textures in MP4 in order to compress them.

然后我需要从我的 Android 上的 MP4 获取它

Then I need to get it from MP4 on my Android

我需要以某种方式解码 MP4 并根据请求逐帧获取.

I need somehow decode MP4 and get frame by frame by request.

我发现了这个MediaCodec

https://developer.android.com/reference/android/media/MediaCodec

和这个 MediaMetadataRetriever

https://developer.android.com/reference/android/media/MediaMetadataRetriever

但是我没有看到如何逐帧请求的方法...

But I did not see approach how to request frame by frame...

如果有人用过 MP4，请告诉我去哪里.

If there is someone who worked with MP4, please give me a way where to go.

P.S. 我正在使用原生方式 (JNI)，所以不管怎么做.. Java 或原生，但我需要找到方式.

P.S. I am working with native way (JNI), so does not matter how to do it.. Java or native, but I need to find the way.

EDIT1

我制作了某种电影(只有一个 3d 模型)，所以我每 32 毫秒更改一次几何图形和纹理.因此，在我看来，将 mp4 用于 tex 似乎是合理的，因为每个新帧(32 毫秒)都与原始帧非常相似......

I make some kind of movie (just one 3d model), so I am changing my geometry as well as textures every 32 milliseconds. So, it is seems to me reasonable to use mp4 for tex because of each new frame (32 milliseconds) very similar to privious one...

现在我为一个模型使用 400 帧.对于几何，我使用 .mtr，对于 tex，我使用 .pkm(因为它针对 android 进行了优化)，所以我有大约 350 个 .mtr 文件(因为有些文件包含子索引)和 400 个 .pkm 文件......

Now I use 400 frames for one model. For geometry I use .mtr and for tex I use .pkm (because it optimized for android) , so I have around 350 .mtr files(because some files include subindex) and 400 .pkm files ...

这就是我将 mp4 用于 tex 的原因.因为一个 mp4 比 400 .pkm 小得多

This is the reason why I am going to use mp4 for tex. Because one mp4 much more smaller than 400 .pkm

EDIT2

请看一下Edit1

实际上我需要知道的只是有可以按帧读取 MP4 的 Android API?也许某种 getNextFrame() 方法?

Actually all that I need to know is there API of Android that could read MP4 by frames? Maybe some kind of getNextFrame() method?

类似的东西

MP4Player player = new MP4Player(PATH_TO_MY_MP4_FILE);

void readMP4(){
   Bitmap b;

   while(player.hasNext()){
      b = player.getNextFrame();

      ///.... my code here ...///
   }
}

EDIT3

我在 Java 上做了这样的实现

I made such implementation on Java

public static void read(@NonNull final Context iC, @NonNull final String iPath)
{
    long time;

    int fileCount = 0;

    //Create a new Media Player
    MediaPlayer mp = MediaPlayer.create(iC, Uri.parse(iPath));
    time = mp.getDuration() * 1000;

    Log.e("TAG", String.format("TIME :: %s", time));

    MediaMetadataRetriever mRetriever = new MediaMetadataRetriever();
    mRetriever.setDataSource(iPath);

    long a = System.nanoTime();

    //frame rate 10.03/sec, 1/10.03 = in microseconds 99700
    for (int i = 99700 ; i <= time ; i = i + 99700)
    {
        Bitmap b = mRetriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);

        if (b == null)
        {
            Log.e("TAG", String.format("BITMAP STATE :: %s", "null"));
        }
        else
        {
            fileCount++;
        }

        long curTime = System.nanoTime();
        Log.e("TAG", String.format("EXECUTION TIME :: %s", curTime - a));
        a = curTime;
    }

    Log.e("TAG", String.format("COUNT :: %s", fileCount));
}

这里是执行时间

  E/TAG: EXECUTION TIME :: 267982039
  E/TAG: EXECUTION TIME :: 222928769
  E/TAG: EXECUTION TIME :: 289899461
  E/TAG: EXECUTION TIME :: 138265423
  E/TAG: EXECUTION TIME :: 127312577
  E/TAG: EXECUTION TIME :: 251179654
  E/TAG: EXECUTION TIME :: 133996500
  E/TAG: EXECUTION TIME :: 289730345
  E/TAG: EXECUTION TIME :: 132158270
  E/TAG: EXECUTION TIME :: 270951461
  E/TAG: EXECUTION TIME :: 116520808
  E/TAG: EXECUTION TIME :: 209071269
  E/TAG: EXECUTION TIME :: 149697230
  E/TAG: EXECUTION TIME :: 138347269

这次以纳秒为单位 == +/- 200 毫秒...它非常缓慢...我需要大约 30 毫秒的帧.

This time in nanoseconds == +/- 200 milliseconds... It is very slowly... I need around 30 milliseconds by frame.

所以，我认为这个方法是在CPU上执行的，所以请问有没有在GPU上执行的方法?

EDIT4

我发现有MediaCodec类

https://developer.android.com/reference/android/media/MediaCodec

我也在这里发现了类似的问题 MediaCodec 从视频中获取所有帧

also I found similar question here MediaCodec get all frames from video

我知道有一种方法可以按字节读取，但不能按帧读取...

I understood that there is a way to read by bytes, but not by frames...

那么，仍然有疑问 - 是否有办法逐帧读取 mp4 视频?

So, still question - if there is a way to read mp4 video by frames?

推荐答案

该解决方案类似于 ExtractMpegFramesTest，其中 MediaCodec 用于从视频帧生成外部"纹理.在测试代​​码中，帧被渲染到屏幕外的 pbuffer，然后保存为 PNG.你可以直接渲染它们.

The solution would look something like the ExtractMpegFramesTest, in which MediaCodec is used to generate "external" textures from video frames. In the test code, the frames are rendered to an off-screen pbuffer and then saved as PNG. You would just render them directly.

这有几个问题:

1. MPEG 视频的设计不能很好地用作随机访问数据库.常见的 GOP(图片组)结构有一个关键帧"(本质上是一个 JPEG 图像)，后跟 14 个增量帧，这些帧仅包含与前一个解码帧的差异.因此，如果您想要第 N 帧，则可能必须首先解码第 N-14 到 N-1 帧.如果您总是向前移动(在纹理上播放电影)或者您只存储关键帧(此时您已经发明了一个笨拙的 JPEG 图像数据库)，这不是问题.
2. 如评论和答案中所述，您可能会看到一些视觉伪影.这些看起来有多糟糕取决于材料和您的压缩率.由于您正在生成帧，因此您可以通过确保在发生重大变化时第一帧始终是关键帧来减少这种情况.
3. MediaCodec 接口的固件可能需要几个帧才能开始生成输出，即使您从关键帧开始也是如此.在流中四处寻找会产生延迟成本.见例如这篇文章.(有没有想过为什么 DVR 快进流畅，快退不流畅?)
4. 通过 SurfaceTexture 的 MediaCodec 帧成为外部"纹理.这些与普通纹理相比有一些限制——性能可能更差，不能用作 FBO 中的颜色缓冲区，等等.如果您只是以 30fps 的速度每帧渲染一次，这应该无关紧要.
由于上述原因，
5. MediaMetadataRetriever 的 getFrameAtTime() 方法的性能不尽如人意.您自己编写它不太可能获得更好的结果，尽管您可以跳过创建 Bitmap 对象的步骤来节省一些时间.此外，您传入了 OPTION_CLOSEST_SYNC，但只有当您的所有帧都是同步帧(同样是笨拙的 JPEG 图像数据库)时，才会产生您想要的结果.您需要使用 OPTION_CLOSEST.

1. MPEG video isn't designed to work well as a random-access database. A common GOP (group of pictures) structure has one "key frame" (essentially a JPEG image) followed by 14 delta frames, which just hold the difference from the previous decoded frame. So if you want frame N, you may have to decode frames N-14 through N-1 first. Not a problem if you're always moving forward (playing a movie onto a texture) or you only store key frames (at which point you've invented a clumsy database of JPEG images).
2. As mentioned in comments and answers, you're likely to get some visual artifacts. How bad these look depends on the material and your compression rate. Since you're generating the frames, you may be able to reduce this by ensuring that, whenever there's a big change, the first frame is always a key frame.
3. The firmware that MediaCodec interfaces with may want several frames before it starts producing output, even if you start at a key frame. Seeking around in a stream has a latency cost. See e.g. this post. (Ever wonder why DVRs have smooth fast-forward, but not smooth fast-backward?)
4. MediaCodec frames passed through SurfaceTexture become "external" textures. These have some limitations vs. normal textures -- performance may be worse, can't use as color buffer in an FBO, etc. If you're just rendering it once per frame at 30fps this shouldn't matter.
5. MediaMetadataRetriever's getFrameAtTime() method has less-than-desirable performance for the reasons noted above. You're unlikely to get better results by writing it yourself, although you can save a bit of time by skipping the step where it creates a Bitmap object. Also, you passed OPTION_CLOSEST_SYNC in, but that will only produce the results you want if all your frames are sync frames (again, clumsy database of JPEG images). You need to use OPTION_CLOSEST.

如果您只是想在纹理上播放电影(或者您的问题可以简化为)，Grafika 有一些例子.一个可能相关的是 TextureFromCamera，它在可以缩放和旋转的 GLES 矩形上呈现相机视频流.您可以使用其他演示之一中的 MP4 播放代码替换相机输入.如果你只是向前玩，这会很好用，但如果你想跳过或向后，你会遇到麻烦.

If you're just trying to play a movie on a texture (or your problem can be reduced to that), Grafika has some examples. One that may be relevant is TextureFromCamera, which renders the camera video stream on a GLES rect that can be zoomed and rotated. You can replace the camera input with the MP4 playback code from one of the other demos. This'll work fine if you're only playing forward, but if you want to skip around or go backward you'll have trouble.

您所描述的问题听起来与 2D 游戏开发人员处理的问题非常相似.做他们所做的可能是最好的方法.

The problem you're describing sounds pretty similar to what 2D game developers deal with. Doing what they do is probably the best approach.

这篇关于如何从 MP4 中逐帧获取?(媒体编解码器)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！