RTMP是否支持h264流中的显示方向SEI消息? [英] Does RTMP support the Display Orientation SEI Message in h264 streams?

问题描述

我正在使用本机MediaCodec API通过Android上的RTMP流传输h264视频和AAC音频.视频和音频看起来很棒，但是当以Potrait模式拍摄视频时，在网络上或使用VLC进行播放始终是横向的.

I'm streaming video h264 video and AAC audio over RTMP on Android using the native MediaCodec APIs. Video and audio look great, however while the video is shot in potrait mode, playback on the web or with VLC is always in landscape.

通读h264规范后，我发现可以在补充增强信息(SEI)中指定这种额外的元数据，并且已经将其添加到原始的h264位流中.我的SEI NAL单元遵循以下基本格式，我计划稍后进行优化:

Having read through the h264 spec, I see that this sort of extra metadata can be specified in Supplemental Enhancement Information (SEI), and I've gone about adding it to the raw h264 bit stream. My SEI NAL unit for this follows this rudimentary format, I plan to optimize later:

val displayOrientationSEI = {
    val prefix = byteArrayOf(0, 0, 0, 1)
    val nalHeader = byteArrayOf(6) // forbidden_zero_bit:0; nal_ref_idc:0; nal_unit_type:6 

    val display = byteArrayOf(47 /* Display orientation type*/, 3 /*payload size*/)

    val displayOrientationCancelFlag = "0" // u(1); Rotation information follows
    val horFlip = "1" // hor_flip; u(1); Flip horizontally
    val verFlip = "1" // ver_flip; u(1); Flip vertically
    val anticlockwiseRotation = "0100000000000000" // u(16); value / 2^16 -> 90 degrees
    val displayOrientationRepetitionPeriod = "010" // ue(v); Persistent till next video sequence
    val displayOrientationExtensionFlag = "0" // u(1); No other value is permitted by the spec atm
    val byteAlignment = "1"

    val bitString = displayOrientationCancelFlag +
            horFlip +
            verFlip +
            anticlockwiseRotation +
            displayOrientationRepetitionPeriod +
            displayOrientationExtensionFlag +
            byteAlignment

    prefix + nalHeader + display + BigInteger(bitString, 2).toByteArray()
}()

使用用于LibRtmp的Android JNI包装器将这些数据包写到RTMP流中a>.

Using Jcodec's SEI class, I can see that my SEI message is parsed properly. I write out these packets to the RTMP stream using an Android JNI wrapper for LibRtmp.

尽管如此，ffprobe不会显示方向元数据，并且播放时的视频仍保持横向.

Despite this, ffprobe does not show the orientation metadata, and the video when played remains in landscape.

在这一点上，我认为我缺少有关LibRtmp写入原始h264单元时FLV头如何工作的非常小的细节.我尝试添加此 displayOrientationSEI NAL单元:

At this point I think I'm missing a very small detail about how FLV headers work when the raw h264 units are written out by LibRtmp. I have tried appending this displayOrientationSEI NAL unit:

1. 仅适用于初始SPS和PPS配置.
2. 直接从编码器获取每个原始的h264 NAL单元.
3. 对双方.

我做错了什么?浏览一些RTMP库的源代码，例如 rtmp-rtsp-stream-client-Java ，似乎在创建FLV标签时会丢弃SEI消息.

What am I doing wrong? Going through the source of some RTMP libraries, like rtmp-rtsp-stream-client-java, it seems the SEI message is dropped when creating FLV tags.

非常感谢帮助.

推荐答案

RTMP是否支持h264流中的显示方向SEI消息?

Does RTMP support the Display Orientation SEI Message in h264 streams?

RTMP并不了解这个概念.从RTMP的角度来看，SEI只是它复制的一系列字节.它从不看它们，也从不解析它们.

RTMP is unaware of the very concept. from RTMPs perspective, the SEI is just a series of bytes it copys. It never looks at them, it never parses them.

需要支持它的是h.264解码器(RTMP也不知道)和播放器软件.如果它不适合您，则必须检查播放器或编码的SEI的有效性，而不是传输.

The thing that needs to support it, is the h.264 decoder (which RTMP is also unaware of) and the player software. If it is not working for you, you must check the player, or the validity of the encoded SEI, not the transport.

这篇关于RTMP是否支持h264流中的显示方向SEI消息?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！