故障同步了libavformat /与ffmpeg的X264和RTP [英] Trouble syncing libavformat/ffmpeg with x264 and RTP
问题描述
我一直对一些流媒体软件，需要活饲料
从各种照相机和使用流通过网络
H.264。要做到这一点，我使用的x264连接直接codeR（用
在zerolatencypreSET）和饲养的NAL，因为他们提供给
libavformat去打包成RTP（最终RTSP）。理想情况下，这
应用应尽可能实时越好。在大多数情况下，
这一直运作良好。
I've been working on some streaming software that takes live feeds from various kinds of cameras and streams over the network using H.264. To accomplish this, I'm using the x264 encoder directly (with the "zerolatency" preset) and feeding NALs as they are available to libavformat to pack into RTP (ultimately RTSP). Ideally, this application should be as real-time as possible. For the most part, this has been working well.
然而不幸的是，有某种同步问题：
在客户端上的任何视频播放似乎显示出一些光滑的框架，
接着是短暂的停顿，那么更多的帧;重复。另外，
似乎有大约4秒的延迟。这种情况与
每一个视频播放器我试过：图腾，VLC和基本的GStreamer管道
Unfortunately, however, there is some sort of synchronization issue: any video playback on clients seems to show a few smooth frames, followed by a short pause, then more frames; repeat. Additionally, there appears to be approximately a 4-second delay. This happens with every video player I've tried: Totem, VLC, and basic gstreamer pipes.
我煮了这一切了几分小测试用例：
I've boiled it all down to a somewhat small test case:
#include <stdio.h>
#include <stdint.h>
#include <unistd.h>
#include <x264.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#define WIDTH 640
#define HEIGHT 480
#define FPS 30
#define BITRATE 400000
#define RTP_ADDRESS "127.0.0.1"
#define RTP_PORT 49990
struct AVFormatContext* avctx;
struct x264_t* encoder;
struct SwsContext* imgctx;
uint8_t test = 0x80;
void create_sample_picture(x264_picture_t* picture)
{
// create a frame to store in
x264_picture_alloc(picture, X264_CSP_I420, WIDTH, HEIGHT);
// fake image generation
// disregard how wrong this is; just writing a quick test
int strides = WIDTH / 8;
uint8_t* data = malloc(WIDTH * HEIGHT * 3);
memset(data, test, WIDTH * HEIGHT * 3);
test = (test << 1) | (test >> (8 - 1));
// scale the image
sws_scale(imgctx, (const uint8_t* const*) &data, &strides, 0, HEIGHT,
picture->img.plane, picture->img.i_stride);
}
int encode_frame(x264_picture_t* picture, x264_nal_t** nals)
{
// encode a frame
x264_picture_t pic_out;
int num_nals;
int frame_size = x264_encoder_encode(encoder, nals, &num_nals, picture, &pic_out);
// ignore bad frames
if (frame_size < 0)
{
return frame_size;
}
return num_nals;
}
void stream_frame(uint8_t* payload, int size)
{
// initalize a packet
AVPacket p;
av_init_packet(&p);
p.data = payload;
p.size = size;
p.stream_index = 0;
p.flags = AV_PKT_FLAG_KEY;
p.pts = AV_NOPTS_VALUE;
p.dts = AV_NOPTS_VALUE;
// send it out
av_interleaved_write_frame(avctx, &p);
}
int main(int argc, char* argv[])
{
// initalize ffmpeg
av_register_all();
// set up image scaler
// (in-width, in-height, in-format, out-width, out-height, out-format, scaling-method, 0, 0, 0)
imgctx = sws_getContext(WIDTH, HEIGHT, PIX_FMT_MONOWHITE,
WIDTH, HEIGHT, PIX_FMT_YUV420P,
SWS_FAST_BILINEAR, NULL, NULL, NULL);
// set up encoder presets
x264_param_t param;
x264_param_default_preset(¶m, "ultrafast", "zerolatency");
param.i_threads = 3;
param.i_width = WIDTH;
param.i_height = HEIGHT;
param.i_fps_num = FPS;
param.i_fps_den = 1;
param.i_keyint_max = FPS;
param.b_intra_refresh = 0;
param.rc.i_bitrate = BITRATE;
param.b_repeat_headers = 1; // whether to repeat headers or write just once
param.b_annexb = 1; // place start codes (1) or sizes (0)
// initalize
x264_param_apply_profile(¶m, "high");
encoder = x264_encoder_open(¶m);
// at this point, x264_encoder_headers can be used, but it has had no effect
// set up streaming context. a lot of error handling has been ommitted
// for brevity, but this should be pretty standard.
avctx = avformat_alloc_context();
struct AVOutputFormat* fmt = av_guess_format("rtp", NULL, NULL);
avctx->oformat = fmt;
snprintf(avctx->filename, sizeof(avctx->filename), "rtp://%s:%d", RTP_ADDRESS, RTP_PORT);
if (url_fopen(&avctx->pb, avctx->filename, URL_WRONLY) < 0)
{
perror("url_fopen failed");
return 1;
}
struct AVStream* stream = av_new_stream(avctx, 1);
// initalize codec
AVCodecContext* c = stream->codec;
c->codec_id = CODEC_ID_H264;
c->codec_type = AVMEDIA_TYPE_VIDEO;
c->flags = CODEC_FLAG_GLOBAL_HEADER;
c->width = WIDTH;
c->height = HEIGHT;
c->time_base.den = FPS;
c->time_base.num = 1;
c->gop_size = FPS;
c->bit_rate = BITRATE;
avctx->flags = AVFMT_FLAG_RTP_HINT;
// write the header
av_write_header(avctx);
// make some frames
for (int frame = 0; frame < 10000; frame++)
{
// create a sample moving frame
x264_picture_t* pic = (x264_picture_t*) malloc(sizeof(x264_picture_t));
create_sample_picture(pic);
// encode the frame
x264_nal_t* nals;
int num_nals = encode_frame(pic, &nals);
if (num_nals < 0)
printf("invalid frame size: %d\n", num_nals);
// send out NALs
for (int i = 0; i < num_nals; i++)
{
stream_frame(nals[i].p_payload, nals[i].i_payload);
}
// free up resources
x264_picture_clean(pic);
free(pic);
// stream at approx 30 fps
printf("frame %d\n", frame);
usleep(33333);
}
return 0;
}
此测试显示在白色背景上的黑色线条的
应平稳地移动到左边。它已被用于的ffmpeg 0.6.5写入
但问题可以被复制的 0.8 和 0.10 （从我到目前为止测试）。我做了一些简略的错误处理，以使这个例子尽可能短
可能同时还显示了问题，所以请原谅一些
讨厌code。我还要指出，虽然一个SDP是不是用在这里，我
使用已经有类似的结果都试过了。该测试可以是
编译：
This test shows black lines on a white background that should move smoothly to the left. It has been written for ffmpeg 0.6.5 but the problem can be reproduced on 0.8 and 0.10 (from what I've tested so far). I've taken some shortcuts in error handling to make this example as short as possible while still showing the problem, so please excuse some of the nasty code. I should also note that while an SDP is not used here, I have tried using that already with similar results. The test can be compiled with:
gcc -g -std=gnu99 streamtest.c -lswscale -lavformat -lx264 -lm -lpthread -o streamtest
它可以与以gtreamer直接播放：
It can be played with gtreamer directly:
gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink
您应该立即注意到口吃。一个常见的修复我有
看到在互联网上是添加同步= false以管道：
You should immediately notice the stuttering. One common "fix" I've seen all over the Internet is to add sync=false to the pipeline:
gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink sync=false
此使得重放为平滑的（和近实时），但是一个
无解，只有在GStreamer工作。我想修复
在源的问题。我已经能够与流近乎相同
使用原始的ffmpeg和参数也没有任何问题：
This causes playback to be smooth (and near-realtime), but is a non-solution and only works with gstreamer. I'd like to fix the problem at the source. I've been able to stream with near-identical parameters using raw ffmpeg and haven't had any issues:
ffmpeg -re -i sample.mp4 -vcodec libx264 -vpre ultrafast -vpre baseline -b 400000 -an -f rtp rtp://127.0.0.1:49990 -an
所以很明显我做错了什么。但究竟是什么？
So clearly I'm doing something wrong. But what is it?
推荐答案
1）您没有设置为PTS帧发送到libx264（你可能会看到非严格单调PTS警告）
2）你没有设置PTS / DTS数据包发送给了libavformat的RTP流合并器（我不是100％肯定它需要设置的，但我想它会更好，从源头code，它看起来像RTP使用PTS） 。
3）恕我直言usleep（33333）是坏的。这导致连接codeR拖延此时也（增加延迟），同时你可以连接code在此期间，即使你还没有需要通过RTP向它发送一个帧。
1) You didn't set PTS for frames you send to libx264 (you probably should see "non-strictly-monotonic PTS" warnings) 2) You didn't set PTS/DTS for packets you send to libavformat's rtp muxer (I not 100% sure it need to be set but I guess it would be better. From source code it looks like rtp use PTS). 3) IMHO usleep(33333) is bad. It cause encoder to stall this time also (increasing latency) while you could encode next frame during this time even if you still don't need to send it by rtp.
P.S。顺便说一句，你没有设置param.rc.i_rc_method为X264_RC_ABR所以libx264将使用CRF 23来代替，而忽略你的param.rc.i_bitrate = BITRATE。此外，它可以编码网络发送时使用VBV好主意。
P.S. btw you didn't set param.rc.i_rc_method to X264_RC_ABR so libx264 will use CRF 23 instead and ignore your "param.rc.i_bitrate = BITRATE". Also it can be good idea to use VBV when encoding for network sending.
这篇关于故障同步了libavformat /与ffmpeg的X264和RTP的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
