不一致的采样时间/ presentation时间在视频解码 [英] Inconsistent sample time / presentation time during video decoding

问题描述

我正在写一个应用程序，可以连接code视频通过摄像头输入和处理视频由德code-编辑-CN code步骤。对于摄像头，我用相机类而不是意图将相机的细节设置。然后我喂相机帧的EN codeR（媒体codeC在API 16）和复用器（我用的ffmpeg复用器，因为我要工作4.1设备）。

I'm writing an APP that can encode video by camera input and process video by decode-edit-encode steps. For the camera, I use the Camera class rather than Intent to configure the details settings of the camera. Then I feed the camera frames to the encoder (MediaCodec in API 16) and the muxer (I use ffmpeg muxer since I want to work on 4.1 Devices).

予测相机帧由系统纳米时间的时间code和选择帧的子集，以适应期望的FPS（目前15）。有在时间值小一些噪音，例如（毫秒）：0，60718，135246，201049，......而不是0，66000，133000，200000 ...

I measure the time code of camera frames by system nano time, and select a subset of frames to fit a desired FPS (currently 15). There are some small "noises" in the time values, for example (in ms): 0, 60718, 135246, 201049, ... rather than 0, 66000, 133000, 200000, ... .

经过一番努力正确地配置复用器的（如<一个href="http://stackoverflow.com/questions/22324990/problems-of-using-media$c$cc-getoutputformat-for-an-en$c$cr-in-android-4-1-4-2">this问题），我可以产生一个视频（与AVC codeC），可以通过在设备上的视频播放器可以播放。播放速度是正确的，所以我觉得影片应该具备的帧的正确时间信息。

After some trying of configuring the muxer correctly (As this question), I can produce a video (with AVC codec) that can be playback by the video player on devices. The playback speed is correct so I think the video should have correct time information of the frames.

不过，我有一个问题，当我尝试去code中的视频进行视频编辑工作流程。我用的是标准的视频提取物/解code步骤这些样品的，就像这样：

However, I got a problem when I try to decode the video to perform the video editting process. I use the standard video extract/decode steps as these samples, like this:

int decode_input_index = decoder.dequeueInputBuffer(TIMEOUT_USEC);
if (decode_input_index >= 0)
{
    ByteBuffer decoder_input_buffer = decode_input_buffers[decode_input_index];
    int sample_size = extractor.readSampleData(decoder_input_buffer, 0);
    if (sample_size < 0)
    {
        decoder.queueInputBuffer(decode_input_index, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
        is_decode_input_done = true;
    }
    else
    {
        long sample_time = extractor.getSampleTime();
        decoder.queueInputBuffer(decode_input_index, 0, sample_size, sample_time, 0);
        extractor.advance();
    }
}
else
{
    Log.v(TAG, "Decoder dequeueInputBuffer timed out! Try again later");
}

从getSampleTime（）采样时间有正确的值，因为我EN code中的视频。 （例如，它们是完全0，60718，135246，201049，......在美国）。它也是在德coder.queueInputBuffer的输入端的presentation时间（）。当去codeR进入脱code这个框架，我用得到的帧时间：

The sample time from getSampleTime() has correct value as I encode the video. (e.g., they are exactly 0, 60718, 135246, 201049, ... in us). It is also the presentation time in the input of decoder.queueInputBuffer(). When the decoder proceeds to decode this frame, I get the frame time by:

int decode_output_index = decoder.dequeueOutputBuffer(decode_buffer_info, TIMEOUT_USEC);
switch (decode_output_index)
{
    ....
    (some negative-value flags in MediaCodec)
    ....
    default:
    {
        ByteBuffer decode_output_buffer = decode_output_buffers[decode_output_index];
        long ptime_us = decode_buffer_info.presentationTimeUs;
        boolean is_decode_EOS = ((decode_buffer_info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0);

        ....
    }
}

我希望设置相同的时间序列，一个在德codeR输入，但我得到了很多的0距离BufferInfo在德codeR输出。去codeD帧内容似乎是正确的，但是大部分presentation时间值是0。只有最后几帧有正确的presentation时间。

I expect to set the same time sequence as the one in decoder input, but I get a lot of 0's from the BufferInfo at decoder output. The decoded frame contents seems to be correct, but most of the presentation time values are 0. Only the last few frames has correct presentation time.

我测试整个过程同样采用Android 4.3（即使采用相同的ffmpeg复用器，而不是MediaMuxer在API 18）的设备上，一切都看起来很好。在4.1 / 4.2设备，如果我拍摄的视频内置相机应用的设备上，然后去code中的视频，那么presentation时间也是正确的，尽管时间值也有因噪音摄像头的延迟。

I test the whole same process on a device with Android 4.3 (even with the same ffmpeg muxer rather than MediaMuxer in API 18), and everything looks fine. On 4.1/4.2 devices, if I capture the video by the built-in camera APP on the device and then decode the video, then the presentation time is also correct although the time values also have noises due to camera delay.

什么是错的视频或去code的过程中，当视频可以和重演德codeD正常，但正确的采样时间和坏presentation时间？我可能不得不使用一种变通方法的采样时间来衡量presentation时间（这很容易通过使用一个队列），但我想弄清楚，如果有任何缺失的部分在我的工作。

What's wrong with the video or the decode process, when the video can be playbacked and decoded normally, but with correct sample time and bad presentation time? I may have to use a workaround to measure the presentation time by the sample time (It's easy by using a queue), but I want to figure out if there is any missing part in my work.

推荐答案

有没有保证正确媒体codeC 处理presentation时间标记的Andr​​oid之前4.3。这是因为确认该PTS行为在CTS测试没有添加到那时

There is no guarantee that MediaCodec handles presentation time stamps correctly before Android 4.3. This is because the CTS tests that confirm the PTS behavior were not added until then.

我不记得有问题的时间戳处理的某些厂商AVC codeCS。我不记得细节副手武器，但是如果你运行的缓冲区到缓冲区和缓冲区到面，从的恩codeDE codeTEST 上的各种4.1 / 4.2设备，你就会把一些故障。 （你需要剥离过程中的出表面到表面测试。）

I do recall that there were problems with the timestamp handling in the AVC codecs from certain vendors. I don't remember the details offhand, but if you run the buffer-to-buffer and buffer-to-surface tests from EncodeDecodeTest on a variety of 4.1/4.2 devices you'll turn up some failures. (You'd need to strip out the surface-to-surface tests, of course.)

您时间戳处理code看起来不错。时间戳不是H.264视频流的一部分，所以它实际上只是正在通过codeC作为元数据转发，你似乎是捡了起来，就转发在所有适当的地方。底线是，如果你在通过有效的PTS值和获得良好的视频，但垃圾PTS值，一些在codeC是处理不当他们。

Your timestamp handling code looks fine. The timestamp isn't part of the H.264 stream, so it's really just being forwarded through the codec as metadata, and you seem to be picking it up and forwarding it on in all the right places. The bottom line is, if you're passing valid PTS values in and getting good video but garbage PTS values, something in the codec is mishandling them.

您将需要解决它通过分别传递的价值观，或者 - 如果输入帧速率始终是常规 - 平凡的计算它。从理论上来说，EN codeR可以重新排列帧，所以在其中传递时间标记到EN codeR的顺序可能是不一样的顺序，他们出来了......但既然你知道了时间戳上升当你做了电影，你应该能够只对它们进行排序，如果这是在实践中存在的问题。

You'll need to work around it by passing the values separately, or -- if the input frame rate is always regular -- trivially computing it. In theory the encoder can reorder frames, so the order in which you pass time stamps into the encoder might not be the same order in which they come out... but since you know the timestamps were ascending when you made the movie, you should be able to just sort them if this were a problem in practice.

在一个不同的音符，在系统的延迟会造成抖动你看到的时间戳值，如果你抓住 System.nanoTime（）的帧时到达应用程序。你可以在安卓4.3好一点与表面输入，因为表面纹理认为，设置更接近当框架被抓获的时间戳。 （我知道这是不是你目前的工作是有用的，但想给一些对未来的希望。）

On a different note, delays in the system will cause the "wobble" you're seeing in the timestamp values if you grab System.nanoTime() when the frame arrives in the app. You can do a bit better in Android 4.3 with Surface input because the SurfaceTexture holds a timestamp that is set much closer to when the frame was captured. (I know that's not useful for your current efforts, but wanted to give some hope for the future.)

这篇关于不一致的采样时间/ presentation时间在视频解码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！