在Android NDK中使用Libavfilter库实现多输入过滤器图 [英] Implementing a multiple input filter graph with the Libavfilter library in Android NDK

问题描述

我正在为Android应用使用带有多个输入源的覆盖过滤器。基本上，我想在静态图像之上覆盖多个视频源。
我查看了ffmpeg附带的示例，并根据这些示例实现了我的代码，但事情似乎没有按预期工作。

ffmpeg过滤样本似乎有一个视频输入。我必须处理多个视频输入，我不知道我的解决方案是正确的。我试图找到其他例子，但是看起来就是这个唯一的一个。

这是我的代码：

< pre class =lang-c prettyprint-override> AVFilterContext ** inputContexts;
AVFilterContext * outputContext;
AVFilterGraph * graph;

int initFilters（AVFrame * bgFrame，int inputCount，AVCodecContext ** codecContexts，char * filters）
{
int i;
int returnCode;
char args [512];
char name [9];
AVFilterInOut ** graphInputs = NULL;
AVFilterInOut * graphOutput = NULL;

AVFilter * bufferSrc = avfilter_get_by_name（buffer）;
AVFilter * bufferSink = avfilter_get_by_name（buffersink）;

graph = avfilter_graph_alloc（）;
if（graph == NULL）
return -1;

//分配输入
graphInputs = av_calloc（inputCount + 1，sizeof（AVFilterInOut *））; （i = 0; i< = inputCount; i ++）
{
graphInputs [i] = avfilter_inout_alloc（）;

if（graphInputs [i] == NULL）
return -1;
}

//分配输入上下文
inputContexts = av_calloc（inputCount + 1，sizeof（AVFilterContext *））;
// first is the background
snprintf（args，sizeof（args），video_size =％dx％d：pix_fmt =％d：time_base = 1/1：pixel_aspect = 0，bgFrame-> ; width，bgFrame-> height，bgFrame->格式）;
returnCode = avfilter_graph_create_filter（& inputContexts [0]，bufferSrc，background，args，NULL，graph）;
if（returnCode< 0）
return returnCode;
graphInputs [0] - > filter_ctx = inputContexts [0];
graphInputs [0] - > name = av_strdup（background）;
graphInputs [0] - > next = graphInputs [1];

//为（i = 1; i <= inputCount; i ++）分配休息

{
AVCodecContext * codecCtx = codecContexts [i - 1] ;
snprintf（args，sizeof（args），video_size =％dx％d：pix_fmt =％d：time_base =％d /％d：pixel_aspect =％d /％d，
codecCtx-> ; width，codecCtx-> height，codecCtx-> pix_fmt，
codecCtx-> time_base.num，codecCtx-> time_base.den，
codecCtx-> sample_aspect_ratio.num，codecCtx-> ; sample_aspect_ratio.den）;
snprintf（name，sizeof（name），video_％d，i）;

returnCode = avfilter_graph_create_filter（& inputContexts [i]，bufferSrc，name，args，NULL，graph）;
if（returnCode< 0）
return returnCode;

graphInputs [i] - > filter_ctx = inputContexts [i];
graphInputs [i] - > name = av_strdup（name）;
graphInputs [i] - > pad_idx = 0;
if（i< inputCount）
{
graphInputs [i] - > next = graphInputs [i + 1];
}
else
{
graphInputs [i] - > next = NULL;
}
}

//分配输出
graphOutput = avfilter_inout_alloc（）;
returnCode = avfilter_graph_create_filter（& outputContext，bufferSink，out，NULL，NULL，graph）;
if（returnCode< 0）
return returnCode;
graphOutput-> filter_ctx = outputContext;
graphOutput-> name = av_strdup（out）;
graphOutput-> next = NULL;
graphOutput-> pad_idx = 0;

returnCode = avfilter_graph_parse_ptr（graph，filters，graphInputs，& graphOutput，NULL）;
if（returnCode< 0）
return returnCode;

returnCode = avfilter_graph_config（graph，NULL）;
return returnCode;

return 0;
}


过滤器参数的函数传递给 avfilter_graph_parse_ptr ，它可以如下所示： [background] scale = 512x512 [base]; [video_1] scale = 256x256 [tmp_1]; [base] [tmp_1] overlay = 0：0 [out]

在调用 avfilter_graph_config之后， 带有警告：
输出板默认与过滤器实例的类型视频背景缓冲区未连接到任何目标并且错误无效参数。

我没有正确地做什么？

编辑：我发现两个问题：

1. 看起来像 avfilter_graph_parse_ptr 的描述有点模糊。 ouputs 参数表示图表的当前输出的列表，在我的情况下，它是 graphInputs 变量，因为这些是缓冲区过滤器的输出。 输入参数表示图表当前输入的列表，在这种情况下，这是 graphOutput 变量，因为它表示 buffersink 过滤器的输入。




2. 我用一个 scale 过滤器和单个输入。似乎 avfilter_graph_parse_ptr 所需的 AVFilterInOut 结构的名称需要 code>。我尝试过不同的版本： in_1 ， in_link_1 。没有一个工作，我没有找到与此有关的任何文件。




所以问题依然存在。如何实现具有多个输入的过滤器图？

解决方案

我已经找到了一个简单的解决方案。
这涉及将 avfilter_graph_parse_ptr 替换为 avfilter_graph_parse2 并添加缓冲区和 buffersink 过滤到过滤器参数 avfilter_graph_parse2 。

因此，在简单的情况下，您有一个背景图像和一个输入视频，过滤器的值参数应该如下所示：

buffer = video_size = 1024x768：pix_fmt = 2：time_base = 1/25：pixel_aspect = 3937/3937 [IN_1]; buffer = video_size = 1920x1080：pix_fmt = 0：time_base = 1/180000：pixel_aspect = 0/1 [in_2]; [in_1] [in_2] overlay = 0：0 [result]; [result] buffersink

avfilter_graph_parse2 将使所有图形连接并初始化所有过滤器。可以从图形本身检索输入缓冲区和输出缓冲区的过滤器上下文。这些用于从过滤器图形中添加/获取框架。

代码的简化版本如下所示：

  AVFilterContext ** inputContexts; 
 AVFilterContext * outputContext; 
 AVFilterGraph * graph; 
 
 int initFilters（AVFrame * bgFrame，int inputCount，AVCodecContext ** codecContexts）
 {
 int i; 
 int returnCode; 
 char filters [1024]; 
 AVFilterInOut * gis = NULL; 
 AVFilterInOut * gos = NULL; 
 
 graph = avfilter_graph_alloc（）; 
 if（graph == NULL）
 {
 printf（Can not allocate filter graph）; 
 return -1; 
} 
 
 //构建过滤器字符串
 // ... 
 
 returnCode = avfilter_graph_parse2（图形，过滤器，& gis，& ; GOS）; 
 if（returnCode< 0）
 {
 cs_printAVError（can not parse graph。，returnCode）; 
 return returnCode; 
} 
 
 returnCode = avfilter_graph_config（graph，NULL）; 
 if（returnCode< 0）
 {
 cs_printAVError（无法配置图），returnCode）; 
 return returnCode; 
} 
 
 //从图中获取过滤器上下文
 
 return 0; 
} 
  

I am trying to use the overlay filter with multiple input sources, for an Android app. Basically, I want to overlay multiple video sources on top of a static image. I have looked at the sample that comes with ffmpeg and implemented my code based on that, but things don't seem to be working as expected.

In the ffmpeg filtering sample there seems to be a single video input. I have to handle multiple video inputs and I am not sure that my solution is the correct one. I have tried to find other examples, but looks like this is the only one.

Here is my code:

AVFilterContext **inputContexts;
AVFilterContext *outputContext;
AVFilterGraph *graph;

int initFilters(AVFrame *bgFrame, int inputCount, AVCodecContext **codecContexts, char *filters)
{
    int i;
    int returnCode;
    char args[512];
    char name[9];
    AVFilterInOut **graphInputs = NULL;
    AVFilterInOut *graphOutput = NULL;

    AVFilter *bufferSrc  = avfilter_get_by_name("buffer");
    AVFilter *bufferSink = avfilter_get_by_name("buffersink");

    graph = avfilter_graph_alloc();
    if(graph == NULL)
        return -1;

    //allocate inputs
    graphInputs = av_calloc(inputCount + 1, sizeof(AVFilterInOut *));
    for(i = 0; i <= inputCount; i++)
    {
        graphInputs[i] = avfilter_inout_alloc();
        if(graphInputs[i] == NULL)
            return -1;
    }

    //allocate input contexts
    inputContexts = av_calloc(inputCount + 1, sizeof(AVFilterContext *));
    //first is the background
    snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=1/1:pixel_aspect=0", bgFrame->width, bgFrame->height, bgFrame->format);
    returnCode = avfilter_graph_create_filter(&inputContexts[0], bufferSrc, "background", args, NULL, graph);
    if(returnCode < 0)
        return returnCode;
    graphInputs[0]->filter_ctx = inputContexts[0];
    graphInputs[0]->name = av_strdup("background");
    graphInputs[0]->next = graphInputs[1];

    //allocate the rest
    for(i = 1; i <= inputCount; i++)
    {
        AVCodecContext *codecCtx = codecContexts[i - 1];
        snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
                    codecCtx->width, codecCtx->height, codecCtx->pix_fmt,
                    codecCtx->time_base.num, codecCtx->time_base.den,
                    codecCtx->sample_aspect_ratio.num, codecCtx->sample_aspect_ratio.den);
        snprintf(name, sizeof(name), "video_%d", i);

        returnCode = avfilter_graph_create_filter(&inputContexts[i], bufferSrc, name, args, NULL, graph);
        if(returnCode < 0)
            return returnCode;

        graphInputs[i]->filter_ctx = inputContexts[i];
        graphInputs[i]->name = av_strdup(name);
        graphInputs[i]->pad_idx = 0;
        if(i < inputCount)
        {
            graphInputs[i]->next = graphInputs[i + 1];
        }
        else
        {
            graphInputs[i]->next = NULL;
        }
    }

    //allocate outputs
    graphOutput = avfilter_inout_alloc();   
    returnCode = avfilter_graph_create_filter(&outputContext, bufferSink, "out", NULL, NULL, graph);
    if(returnCode < 0)
        return returnCode;
    graphOutput->filter_ctx = outputContext;
    graphOutput->name = av_strdup("out");
    graphOutput->next = NULL;
    graphOutput->pad_idx = 0;

    returnCode = avfilter_graph_parse_ptr(graph, filters, graphInputs, &graphOutput, NULL);
    if(returnCode < 0)
        return returnCode;

    returnCode = avfilter_graph_config(graph, NULL);
        return returnCode;

    return 0;
}

The filters argument of the function is passed on to avfilter_graph_parse_ptr and it can looks like this: [background] scale=512x512 [base]; [video_1] scale=256x256 [tmp_1]; [base][tmp_1] overlay=0:0 [out]

The call breaks after the call to avfilter_graph_config with the warning: Output pad "default" with type video of the filter instance "background" of buffer not connected to any destination and the error Invalid argument.

What is it that I am not doing correctly?

EDIT: The are two issues that I have discovered:

1. Looks like the description of avfilter_graph_parse_ptr is a bit vague. The ouputs parameter represents a list of the current outputs of the graph, in my case that being the graphInputs variable, because these are the outputs from the buffer filter. The inputs parameter represents a list of the current inputs of the graph, in this case this is the graphOutput variable, because it represents the input to the buffersink filter.

2. I did some testing with a scale filter and a single input. It seems that the name of the AVFilterInOut structure required by avfilter_graph_parse_ptr needs to be in. I have tried with different versions: in_1, in_link_1. None of them work and I have not been able to find any documentation related to this.

So the issue still remains. How do I implement a filter graph with multiple inputs?

解决方案

I have found a simple solution to the problem. This involves replacing the avfilter_graph_parse_ptr with avfilter_graph_parse2 and adding the buffer and buffersink filters to the filters parameter of avfilter_graph_parse2.

So, in the simple case where you have one background image and one input video the value of the filters parameter should look like this:

buffer=video_size=1024x768:pix_fmt=2:time_base=1/25:pixel_aspect=3937/3937 [in_1]; buffer=video_size=1920x1080:pix_fmt=0:time_base=1/180000:pixel_aspect=0/1 [in_2]; [in_1] [in_2] overlay=0:0 [result]; [result] buffersink

The avfilter_graph_parse2 will make all the graph connections and initialize all the filters. The filter contexts for the input buffers and for the output buffer can be retrieved from the graph itself at the end. These are used to add/get frames from the filter graph.

A simplified version of the code looks like this:

AVFilterContext **inputContexts;
AVFilterContext *outputContext;
AVFilterGraph *graph;

int initFilters(AVFrame *bgFrame, int inputCount, AVCodecContext **codecContexts)
{
    int i;
    int returnCode;
    char filters[1024];
    AVFilterInOut *gis = NULL;
    AVFilterInOut *gos = NULL;

    graph = avfilter_graph_alloc();
    if(graph == NULL)
    {
        printf("Cannot allocate filter graph.");        
        return -1;
    }

    //build the filters string here
    // ...

    returnCode = avfilter_graph_parse2(graph, filters, &gis, &gos);
    if(returnCode < 0)
    {
        cs_printAVError("Cannot parse graph.", returnCode);
        return returnCode;
    }

    returnCode = avfilter_graph_config(graph, NULL);
    if(returnCode < 0)
    {
        cs_printAVError("Cannot configure graph.", returnCode);
        return returnCode;
    }

    //get the filter contexts from the graph here

    return 0;
}

这篇关于在Android NDK中使用Libavfilter库实现多输入过滤器图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！