从RTP数据包重建图像 [英] Reconstruct image from RTP packets

问题描述

我正在尝试通过网络将用户的网络摄像头流式传输到基于C的服务器.我使用了 Janus网关.

我创建了一个小插件，该插件很大程度上基于echotest演示示例:我的浏览器通过WebRTC技术连接到我的janus服务器，并且流了用户的网络摄像头.

在服务器端，我具有 janus_incomming_rtp 函数，该函数为我提供了 char *缓冲区和 int长度.经过检查，传入的数据缓冲区大约是MTU的长度:我的视频的每一帧都是通过几个RTP数据包发送的.

我已经按照此Wikipedia页面进行了一些检查，但我不知道如何从该UDP RTP数据包流中重建图像.理想情况下，我想将流传递给openCV进行实时图像处理.

我听说过gstreamer，但是我不了解它是什么，也不知道它如何对我有所帮助；除了我不知道openCV是否有任何内置功能来重建"图像?而且我不知道视频帧是以哪种格式编码的: PT (有效负载类型)似乎是116，被定义为动态"，但我不知道它是什么意思./p>

有帮助吗?

解决方案

以下是一些处理SRTP数据包以对其进行解码的指导步骤.

1. 确保未复用rtp和RTCP，可以从SDP中删除该选项
2. 将SRTP数据包解密为原始RTP，您将需要访问密钥交换(不确定是否已经在执行此操作，但是所有媒体都已加密并且使用DTLS交换了密钥，并且在处理之前必须将其解密)
3. 获取您的媒体有效负载类型并将其与SDP中的媒体进行匹配(您可以从SDP中的RTPMAP中看到什么媒体是什么有效负载)
4. 从数据包中删除RTP有效负载(Gstreamer具有用于大多数常见有效负载(包括VP8)的RtpDepay插件)并对流进行解码. 使用vp8的命令行管道的简单示例
5. 现在您可以显示原始的视频/音频数据包了.

SDP:

• 如果要对RTCP和RTP进行多路复用，则会看到该行 a=rtcp-mux，您将看到a=rtcp:50111 IN IP4 <address>中的端口和候选媒体端口相同.
• 如果媒体本身正在多路传输，您将看到a=group:BUNDLE audio video

SRTP:

• Janus已经处理了DTLS交换，并且看起来它可能已经在发送之前对rtp进行了解密，但是看起来它并不能解决rtp/rtcp和媒体的复用问题.
• 此处是一种快速而肮脏的SRTP解密器，当您将其传递给在DTLS中交换.

GStreamer:

• 您可能需要调查 GstAppSrc 这允许您将数组char放入gstreamer管道中，以用于 解码，您可以将其推送到另一个udp端口以使用 OpenCV.
• 这是一些示例 代码 从我写的一个websocket服务器上，它将获取原始媒体并将其推送 到管道.这个例子并不完全是您想要做的(它是 不抓取RTP，而是从网页获取原始媒体帧) 但是它将向您展示如何使用AppSrc.

I am trying to stream a user's webcam over the network to a C-based server. I have used Janus gateway.

I created a small plugin which is heavily based on the echotest demo example : I have my browser connecting to my janus server via WebRTC technology and I have it stream the user's webcam.

On the server side, I have janus_incomming_rtp function which gives me a char * buffer and int length. Upon inspection, the buffer of data that's incomming is about the length of the MTU : each frame of my video is sent upon several RTP packets.

I have inspected a bit the header by following this wikipedia page but I don't know how to reconstruct the image from that stream of UDP RTP packets. Ideally, I'd like to pass the stream to openCV to do realtime image processing.

I have heard of gstreamer, but I don't undertstand what it is nor how it could help me ; besides I don't know if openCV has any built in functions to "reconstruct" the images ? And I don't know in which format the video frames are being encoded : the PT (Payload Type) seems to be 116 which is defined as "dynamic" but I have no idea what it means.

Any help ?

解决方案

Here are some guiding steps for handling the SRTP packets to decode them.

1. Make sure that rtp and RTCP are not multiplexed, you can remove that option from the SDP
2. Decrypt the SRTP packet to raw RTP, you will need access to the key exchange(not sure if you are already doing this but all media is encrypted and keys exchanged using DTLS and must be decrypted before handling)
3. Grab your media payload type and match it against the media from SDP(you can see from the RTPMAP in the SDP what media is what payload)
4. Remove the RTP Payload from the packet(Gstreamer has RtpDepay plugins for most common payloads, including VP8) and decode the stream. Quick example of command line pipelines using vp8
5. Now you have a raw video/audio packet that can be displayed.

SDP:

• If RTCP and RTP are being multiplexed you will see the line a=rtcp-mux and you will see that the port in a=rtcp:50111 IN IP4 <address> and the candidate media ports will be the same.
• If the media itself is being multiplexed you will see a=group:BUNDLE audio video

SRTP:

• Janus handles the DTLS exchange already and it seems that it may already decrypt the rtp before sending it but it does not look like it accounts for multiplexed rtp/rtcp and media.
• Here is a quick and dirty SRTP decrypter that works when you pass it the MasterKey that is exchanged in DTLS.

GStreamer:

• You may want to look into the GstAppSrc which allows you to char arrays into a gstreamer pipeline for decoding and you can push it to another udp port to grab it with OpenCV.
• Here is some example code from a websocket server I wrote that will grab raw media and push it to a pipeline. This example is not exactly what you want to do(it does not grab the RTP but instead raw media frames from the webpage) but it will show you how to use AppSrc.

这篇关于从RTP数据包重建图像的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！