如何从MP4逐帧获取? (MediaCodec) [英] How to get frame by frame from MP4? (MediaCodec)

问题描述

实际上，我正在使用OpenGL，我想将所有纹理都压缩到MP4中以对其进行压缩.

Actually I am working with OpenGL and I would like to put all my textures in MP4 in order to compress them.

然后我需要从Android上的MP4上获取它

Then I need to get it from MP4 on my Android

我需要以某种方式解码MP4并按请求逐帧获取.

I need somehow decode MP4 and get frame by frame by request.

我找到了这个MediaCodec

https://developer.android.com/reference/android/media/MediaCodec

和这个MediaMetadataRetriever

https://developer.android.com/reference/android/media/MediaMetadataRetriever

但是我没有看到如何逐帧请求方法...

But I did not see approach how to request frame by frame...

如果有人使用过MP4，请给我一个方法.

If there is someone who worked with MP4, please give me a way where to go.

P.S..我正在使用本机方式(JNI)，所以如何操作无关紧要..Java或本机，但是我需要找到方式.

P.S. I am working with native way (JNI), so does not matter how to do it.. Java or native, but I need to find the way.

EDIT1

我制作某种电影(仅是一个3D模型)，所以我每32毫秒更改一次几何图形和纹理.因此，在我看来，将mp4用于tex是合理的，因为每个新帧(32毫秒)都与私有帧非常相似...

I make some kind of movie (just one 3d model), so I am changing my geometry as well as textures every 32 milliseconds. So, it is seems to me reasonable to use mp4 for tex because of each new frame (32 milliseconds) very similar to privious one...

现在我为一个模型使用400帧.对于几何图形，我使用.mtr，对于tex，我使用.pkm(因为它已针对android优化)，所以我大约有350 .mtr文件(因为某些文件包含子索引)和400 .pkm文件...

Now I use 400 frames for one model. For geometry I use .mtr and for tex I use .pkm (because it optimized for android) , so I have around 350 .mtr files(because some files include subindex) and 400 .pkm files ...

这就是为什么我要在tex上使用mp4的原因.因为一个mp4比400 .pkm

This is the reason why I am going to use mp4 for tex. Because one mp4 much more smaller than 400 .pkm

EDIT2

请看看Edit1

Plase take a look at Edit1

实际上我只需要知道有Android的API可以按帧读取MP4吗?也许某种getNextFrame()方法?

Actually all that I need to know is there API of Android that could read MP4 by frames? Maybe some kind of getNextFrame() method?

类似的东西

MP4Player player = new MP4Player(PATH_TO_MY_MP4_FILE);

void readMP4(){
   Bitmap b;

   while(player.hasNext()){
      b = player.getNextFrame();

      ///.... my code here ...///
   }
}

EDIT3

我在Java上实现了这样的实现

I made such implementation on Java

public static void read(@NonNull final Context iC, @NonNull final String iPath)
{
    long time;

    int fileCount = 0;

    //Create a new Media Player
    MediaPlayer mp = MediaPlayer.create(iC, Uri.parse(iPath));
    time = mp.getDuration() * 1000;

    Log.e("TAG", String.format("TIME :: %s", time));

    MediaMetadataRetriever mRetriever = new MediaMetadataRetriever();
    mRetriever.setDataSource(iPath);

    long a = System.nanoTime();

    //frame rate 10.03/sec, 1/10.03 = in microseconds 99700
    for (int i = 99700 ; i <= time ; i = i + 99700)
    {
        Bitmap b = mRetriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);

        if (b == null)
        {
            Log.e("TAG", String.format("BITMAP STATE :: %s", "null"));
        }
        else
        {
            fileCount++;
        }

        long curTime = System.nanoTime();
        Log.e("TAG", String.format("EXECUTION TIME :: %s", curTime - a));
        a = curTime;
    }

    Log.e("TAG", String.format("COUNT :: %s", fileCount));
}

这里是执行时间

  E/TAG: EXECUTION TIME :: 267982039
  E/TAG: EXECUTION TIME :: 222928769
  E/TAG: EXECUTION TIME :: 289899461
  E/TAG: EXECUTION TIME :: 138265423
  E/TAG: EXECUTION TIME :: 127312577
  E/TAG: EXECUTION TIME :: 251179654
  E/TAG: EXECUTION TIME :: 133996500
  E/TAG: EXECUTION TIME :: 289730345
  E/TAG: EXECUTION TIME :: 132158270
  E/TAG: EXECUTION TIME :: 270951461
  E/TAG: EXECUTION TIME :: 116520808
  E/TAG: EXECUTION TIME :: 209071269
  E/TAG: EXECUTION TIME :: 149697230
  E/TAG: EXECUTION TIME :: 138347269

这一次以纳秒为单位== +/- 200毫秒...这是非常缓慢的...我需要每帧大约30毫秒.

This time in nanoseconds == +/- 200 milliseconds... It is very slowly... I need around 30 milliseconds by frame.

所以，我认为该方法是在CPU上执行的，所以请问是否有一种方法可以在GPU上执行?

EDIT4

我发现有MediaCodec类

https://developer.android.com/reference/android/media/MediaCodec

我也在这里 MediaCodec从视频中获取所有帧

我知道有一种方法可以按字节读取，但不能按帧读取...

I understood that there is a way to read by bytes, but not by frames...

所以，仍然有一个问题-是否可以通过帧读取mp4视频?

So, still question - if there is a way to read mp4 video by frames?

推荐答案

解决方案类似于 ExtractMpegFramesTest ，其中MediaCodec用于从视频帧生成外部"纹理.在测试代​​码中，将帧渲染到屏幕外的pbuffer中，然后另存为PNG.您只需直接渲染它们即可.

The solution would look something like the ExtractMpegFramesTest, in which MediaCodec is used to generate "external" textures from video frames. In the test code, the frames are rendered to an off-screen pbuffer and then saved as PNG. You would just render them directly.

这有一些问题:

MPEG视频的设计不能很好地用作随机访问数据库. 常见的GOP(图片组)结构具有一个关键帧"(本质上是JPEG图像)，后跟14个增量帧，它们仅保留与先前解码帧的差异.因此，如果要帧N，则可能必须先对帧N-14至N-1进行解码.如果您一直向前移动(将电影播放到纹理上)或仅存储关键帧(此时您已经创建了笨拙的JPEG图像数据库)，这不是问题.
1. 如评论和答案中所述，您可能会看到一些视觉瑕疵.这些外观有多糟糕取决于材料和您的压缩率.由于正在生成帧，因此可以通过确保每当有较大变化时，第一帧始终是关键帧来减少此情况.
2. 即使您从关键帧开始，MediaCodec与之交互的固件在开始产生输出之前可能需要几个帧.在流中四处寻找具有延迟成本.参见例如这篇文章. (是否想过为什么DVR会有顺畅的快进，却没有顺畅的快退?)
通过表面纹理传递的
3. MediaCodec框架成为外部"纹理.这些与普通纹理相比有一些限制-性能可能会更差，不能在FBO中用作颜色缓冲，等等.如果您仅以30fps的速度每帧渲染一次，那就没关系了.
由于上述原因，
4. MediaMetadataRetriever的getFrameAtTime()方法的性能不理想.尽管可以通过跳过创建Bitmap对象的步骤来节省一些时间，但是您自己编写它不太可能获得更好的结果.另外，您传入了OPTION_CLOSEST_SYNC，但是如果所有帧都是同步帧(同样，JPEG图像的笨拙数据库)，则只会产生所需的结果.您需要使用OPTION_CLOSEST.

1. MPEG video isn't designed to work well as a random-access database. A common GOP (group of pictures) structure has one "key frame" (essentially a JPEG image) followed by 14 delta frames, which just hold the difference from the previous decoded frame. So if you want frame N, you may have to decode frames N-14 through N-1 first. Not a problem if you're always moving forward (playing a movie onto a texture) or you only store key frames (at which point you've invented a clumsy database of JPEG images).
2. As mentioned in comments and answers, you're likely to get some visual artifacts. How bad these look depends on the material and your compression rate. Since you're generating the frames, you may be able to reduce this by ensuring that, whenever there's a big change, the first frame is always a key frame.
3. The firmware that MediaCodec interfaces with may want several frames before it starts producing output, even if you start at a key frame. Seeking around in a stream has a latency cost. See e.g. this post. (Ever wonder why DVRs have smooth fast-forward, but not smooth fast-backward?)
4. MediaCodec frames passed through SurfaceTexture become "external" textures. These have some limitations vs. normal textures -- performance may be worse, can't use as color buffer in an FBO, etc. If you're just rendering it once per frame at 30fps this shouldn't matter.
5. MediaMetadataRetriever's getFrameAtTime() method has less-than-desirable performance for the reasons noted above. You're unlikely to get better results by writing it yourself, although you can save a bit of time by skipping the step where it creates a Bitmap object. Also, you passed OPTION_CLOSEST_SYNC in, but that will only produce the results you want if all your frames are sync frames (again, clumsy database of JPEG images). You need to use OPTION_CLOSEST.

如果您只是想在纹理上播放电影(或者您的问题可以解决)，请 Grafika 有一些例子.可能相关的一个是TextureFromCamera，它在可缩放和旋转的GLES矩形上渲染摄像机视频流.您可以使用其他演示之一中的MP4播放代码替换相机输入.如果您只是在播放，这会很好用，但是如果您想跳过或后退，则会遇到麻烦.

If you're just trying to play a movie on a texture (or your problem can be reduced to that), Grafika has some examples. One that may be relevant is TextureFromCamera, which renders the camera video stream on a GLES rect that can be zoomed and rotated. You can replace the camera input with the MP4 playback code from one of the other demos. This'll work fine if you're only playing forward, but if you want to skip around or go backward you'll have trouble.

您所描述的问题听起来与2D游戏开发人员所处理的问题非常相似.做他们可能做的最好的方法.

The problem you're describing sounds pretty similar to what 2D game developers deal with. Doing what they do is probably the best approach.

这篇关于如何从MP4逐帧获取? (MediaCodec)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！