FFmpeg4.0 的filter机制的架构与实现.之三 Filter实现的源码分析

五、Filter实现的源码分析

5.1 滤镜的回调函数的调用流程，以单滤镜设置(如 -vf "delogo")为例

init()

query_format(); // 输入输出的格式查询： 列出滤镜支持的格式列表

config_input();

config_output();

for (;;) {

request_frame();

filter_frame();

}

uninit();

5.2 函数的调用堆栈

STEP1:　　init();                           // 滤镜初始化    ： 用于初始化滤镜
#0  init (ctx=0x24dbe40) at libavfilter/vf_delogo.c:196
#1  0x00000000004e9ad5 in avfilter_init_str (filter=0x24dbe40, args=0x24dbdc0 "x=10:y=10:w=80:h=20") at libavfilter/avfilter.c:1018
#2  0x00000000004fb54a in create_filter (filt_ctx=0x7fffffffce20, ctx=0x24da800, index=<value optimized out>, name=<value optimized out>, args=0x24dbdc0 "x=10:y=10:w=80:h=20", 
    log_ctx=0x24da800) at libavfilter/graphparser.c:149
#3  0x00000000004fbee2 in parse_filter (graph=0x24da800, filters=0x23a5488 "", inputs=0x7fffffffd0b8, outputs=0x7fffffffd0b0) at libavfilter/graphparser.c:192
#4  avfilter_graph_parse2 (graph=0x24da800, filters=0x23a5488 "", inputs=0x7fffffffd0b8, outputs=0x7fffffffd0b0) at libavfilter/graphparser.c:427
#5  0x00000000004bc528 in configure_filtergraph (fg=0x23a5500) at fftools/ffmpeg_filter.c:1047
#6  0x00000000004cb4ba in ifilter_send_frame (ifilter=0x23aabc0, frame=0x24b2480) at fftools/ffmpeg.c:2173
#7  0x00000000004cb9b6 in send_frame_to_filters (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, 
    eof=<value optimized out>, decode_failed=0x7fffffffd578) at fftools/ffmpeg.c:2254
#8  decode_video (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, eof=<value optimized out>, decode_failed=0x7fffffffd578)
    at fftools/ffmpeg.c:2455
#9  0x00000000004cc1ad in process_input_packet (ist=0x23adfc0, pkt=<value optimized out>, no_eof=0) at fftools/ffmpeg.c:2609
#10 0x00000000004ce659 in process_input () at fftools/ffmpeg.c:4441
#11 transcode_step () at fftools/ffmpeg.c:4561
#12 0x00000000004d0e65 in transcode (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4615
#13 main (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4847
preinit();                      // 滤镜预初始化： 用于分配空间和初始化子对象


STEP２:　　query_format();           //  输入输出的格式查询： 列出滤镜支持的格式列表
#0  query_formats (ctx=0x24dbe40) at libavfilter/vf_delogo.c:188
#1  0x00000000004ec8e6 in filter_query_formats (ctx=0x24dbe40) at libavfilter/avfiltergraph.c:326
#2  0x00000000004ed014 in query_formats (graph=0x24da800, log_ctx=0x0) at libavfilter/avfiltergraph.c:454
#3  0x00000000004edc51 in graph_config_formats (graphctx=0x24da800, log_ctx=0x0) at libavfilter/avfiltergraph.c:1163
#4  avfilter_graph_config (graphctx=0x24da800, log_ctx=0x0) at libavfilter/avfiltergraph.c:1274
#5  0x00000000004bcabb in configure_filtergraph (fg=0x23a5500) at fftools/ffmpeg_filter.c:1100
#6  0x00000000004cb4ba in ifilter_send_frame (ifilter=0x23aabc0, frame=0x24b2480) at fftools/ffmpeg.c:2173
#7  0x00000000004cb9b6 in send_frame_to_filters (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, 
    eof=<value optimized out>, decode_failed=0x7fffffffd578) at fftools/ffmpeg.c:2254
#8  decode_video (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, eof=<value optimized out>, decode_failed=0x7fffffffd578)
    at fftools/ffmpeg.c:2455
#9  0x00000000004cc1ad in process_input_packet (ist=0x23adfc0, pkt=<value optimized out>, no_eof=0) at fftools/ffmpeg.c:2609
#10 0x00000000004ce659 in process_input () at fftools/ffmpeg.c:4441
#11 transcode_step () at fftools/ffmpeg.c:4561
#12 0x00000000004d0e65 in transcode (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4615
#13 main (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4847


STEP３:  配置滤镜的输入、输出格式和参数
              　　config_input();
　　　　　　config_output();
#0  config_input (inlink=0x24dd980) at libavfilter/vf_delogo.c:231
#1  0x00000000004ea930 in avfilter_config_links (filter=0x24dbe40) at libavfilter/avfilter.c:369
#2  0x00000000004ea8b0 in avfilter_config_links (filter=0x24da900) at libavfilter/avfilter.c:307
#3  0x00000000004ea8b0 in avfilter_config_links (filter=0x24dec40) at libavfilter/avfilter.c:307
#4  0x00000000004ee922 in graph_config_links (graphctx=0x24da800, log_ctx=0x0) at libavfilter/avfiltergraph.c:261
#5  avfilter_graph_config (graphctx=0x24da800, log_ctx=0x0) at libavfilter/avfiltergraph.c:1276
#6  0x00000000004bcabb in configure_filtergraph (fg=0x23a5500) at fftools/ffmpeg_filter.c:1100
#7  0x00000000004cb4ba in ifilter_send_frame (ifilter=0x23aabc0, frame=0x24b2480) at fftools/ffmpeg.c:2173
#8  0x00000000004cb9b6 in send_frame_to_filters (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570,   eof=<value optimized out>, decode_failed=0x7fffffffd578) at fftools/ffmpeg.c:2254
#9  decode_video (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, eof=<value optimized out>, decode_failed=0x7fffffffd578)  at fftools/ffmpeg.c:2455
#10 0x00000000004cc1ad in process_input_packet (ist=0x23adfc0, pkt=<value optimized out>, no_eof=0) at fftools/ffmpeg.c:2609
#11 0x00000000004ce659 in process_input () at fftools/ffmpeg.c:4441
#12 transcode_step () at fftools/ffmpeg.c:4561
#13 0x00000000004d0e65 in transcode (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4615
#14 main (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4847


STEP４:  滤镜体对每一帧数据进行处理
         　　filter_frame();
#0  filter_frame (inlink=0x24dd980, in=0x24de480) at libavfilter/vf_delogo.c:245
#1  0x00000000004eb92e in ff_filter_frame_framed (filter=<value optimized out>) at libavfilter/avfilter.c:1071
#2  ff_filter_frame_to_filter (filter=<value optimized out>) at libavfilter/avfilter.c:1219
#3  ff_filter_activate_default (filter=<value optimized out>) at libavfilter/avfilter.c:1268
#4  ff_filter_activate (filter=<value optimized out>) at libavfilter/avfilter.c:1430
#5  0x00000000004efad0 in push_frame (ctx=<value optimized out>, frame=0x24b2480, flags=<value optimized out>) at libavfilter/buffersrc.c:181
#6  av_buffersrc_add_frame_internal (ctx=<value optimized out>, frame=0x24b2480, flags=<value optimized out>) at libavfilter/buffersrc.c:255
#7  0x00000000004efc2f in av_buffersrc_add_frame_flags (ctx=0x24ddcc0, frame=<value optimized out>, flags=<value optimized out>) at libavfilter/buffersrc.c:164
#8  0x00000000004cb39d in ifilter_send_frame (ifilter=0x23aabc0, frame=0x24b2480) at fftools/ffmpeg.c:2180
#9  0x00000000004cb9b6 in send_frame_to_filters (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570,  eof=<value optimized out>, decode_failed=0x7fffffffd578) at fftools/ffmpeg.c:2254
#10 decode_video (ist=0x23adfc0, pkt=<value optimized out>, got_output=<value optimized out>, duration_pts=0x7fffffffd570, eof=<value optimized out>, decode_failed=0x7fffffffd578)  at fftools/ffmpeg.c:2455
#11 0x00000000004cc1ad in process_input_packet (ist=0x23adfc0, pkt=<value optimized out>, no_eof=0) at fftools/ffmpeg.c:2609
#12 0x00000000004ce659 in process_input () at fftools/ffmpeg.c:4441
#13 transcode_step () at fftools/ffmpeg.c:4561
#14 0x00000000004d0e65 in transcode (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4615
#15 main (argc=<value optimized out>, argv=<value optimized out>) at fftools/ffmpeg.c:4847

STEP５:　　uninit()

5.3 filter_frame()调用流程

5.3.1. decode_video //ffmpeg.c
最初的源头，是ffmpeg.c的decode_video函数。 
核心代码如下所示：
static int decode_video(InputStream *ist, AVPacket *pkt, int *got_output)
{
    AVFrame* decoded_frame, f;
    //解码
    ret = avcodec_decode_video2(ist->dec_ctx,
                                decoded_frame, got_output, pkt);
    //......
    //将解码后的帧数据  送给滤镜
    for (i = 0; i < ist->nb_filters; i++) {
        f = decodec_frame;
        ret = av_buffersrc_add_frame_flags(ist->filters[i]->filter, f, AV_BUFFERSRC_FLAG_PUSH);
    }
}

可见，最重要做2件事，一个解码，一个送给滤镜。 
送给哪个滤镜呢？
InputStream *ist的nb_filters为1，其实就是指向名字为“buffer”的filter(源文件:buffersrc.c)。
这个filter与其他filter不同的是，它是所有filter的第一个入口。
解码完，都先给它，它再传递给下一个。为啥先给他呢？
很简单，它是一个FIFO，缓存数据用的。

5.3.2. av_buffersrc_add_frame_flags//buffersrc.c
该函数直接走到av_buffersrc_add_frame_internal //buffersrc.c

5.3.3. av_buffersrc_add_frame_internal //buffersrc.c
static int av_buffersrc_add_frame_internal(AVFilterContext *ctx,
                                           AVFrame *frame, int flags)
{
    AVFrame *copy = NULL;
    av_frame_ref(copy, frame);

    // 将解码后的帧数据 frame 写FIFO
    av_fifo_generic_write(s->fifo, &copy, sizeof(copy), NULL);

    // 
    if ((flags & AV_BUFFERSRC_FLAG_PUSH))
        if ((ret = ctx->output_pads[0].request_frame(ctx->outputs[0])) < 0)
            return ret;

    return 0;
}

可见， 该函数把frame写到FIFO，然后调了自己的output_pads[0]的request_frame。

5.3.4. request_frame //buffersrc.c
static int request_frame(AVFilterLink *link)
{
    BufferSourceContext *c = link->src->priv;
    AVFrame *frame;
    int ret;

    //省略......

    // 从FIFO中读取一帧数据 
    av_fifo_generic_read(c->fifo, &frame, sizeof(frame), NULL);
    av_log(NULL, AV_LOG_WARNING, "request_frame, frame-pts %lld \n", frame->pts);

    //这个link，是第一个link，链接当前的AVFilterContext和下一个AVFilterContext，也就用户设置的真正要用的滤镜代码
    ret = ff_filter_frame(link, frame);

    return ret;
}

可见它从FIFO读取一帧数据。然后调用ff_filter_frame。此时输入的link是第一个AVFilterLink。

5.3.5. ff_filter_frame // avfilter.c
该函数做了一些基本检查，走到ff_filter_frame_framed

5.3.6. ff_filter_frame_framed //avfilter.c
{
    //定义一个函数指针filter_frame。所指向的函数，参数为AVFilterLink *, AVFrame *,返回值为int
    int (*filter_frame)(AVFilterLink *, AVFrame *);

    //下一个AVFilterContext，对本例来说，就是我们自己写的transform 滤镜，源码在vf_transform.c
    AVFilterContext *dstctx = link->dst;   
    AVFilterPad *dst = link->dstpad;
    AVFrame *out = NULL;
    int ret;

    //函数指针filter_frame，link->dstpad其实就是dstctx->input_pads，也就是transform滤镜定义的
    if (!(filter_frame = dst->filter_frame))  
        filter_frame = default_filter_frame;
    //省略300字

    ret = filter_frame(link, out);
    link->frame_count++;
    ff_update_link_current_pts(link, pts);
    return ret;
}

定义一个函数指针filter_frame。
所指向的函数，必须是参数为(AVFilterLink , AVFrame ) ,返回值为int 
filter_frame = dst->filter_frame 
dst = link->dstpad，而link->dstpad就是transform过滤器定义的input_pads

static const AVFilterPad avfilter_vf_transform_inputs[] = {
    {
        .name         = "default",
        .type         = AVMEDIA_TYPE_VIDEO,
        .filter_frame = filter_frame,
    },
    { NULL }
};

所以，filter_frame函数指针，指向的就是vf_transform.c实现的filter_frame函数。

5.3.7. filter_frame //vf_transform.c，
当然啦，ffmpeg定义的各种filter，比如vf_colorbalance.c，vf_scale.c等，
也有这个函数，流程一样的
static int filter_frame(AVFilterLink *link, AVFrame *in)
{
    AVFilterContext *avctx = link->dst;             //第一个link的dst AVFilterContext，其实就是当前的filter的AVFilterContext
    AVFilterLink *outlink = avctx->outputs[0];   //当前的AVFilterContext，outputs[0]指向第二个AVFilterLink
    AVFrame *out;

    //分配一个空的AVFrame。
    out = ff_get_video_buffer(outlink, outlink->w, outlink->h);
    if (!out) {
        av_frame_free(&in);
        return AVERROR(ENOMEM);
    }

    //分配的空buffer的参数和上一个基本一致，但修改宽高。当然啦，如果你愿意，不修改宽高，那就不需要下面2句。
    av_frame_copy_props(out, in);
    out->width  = outlink->w;
    out->height = outlink->h;
    out->format = outlink->format;

    ThreadData td;
    td.in = in;
    td.out = out;
    int res;
    if(res = avctx->internal->execute(avctx, do_conversion, &td, NULL, FFMIN(outlink->h, avctx->graph->nb_threads))) {
        return res;
    }//启用一个子线程，执行比较耗时的变换。do_conversion是我们要做的变换。

    av_frame_free(&in);
    
    // 此时的ff_filter_frame，输入参数和前面buffersrc.c调用的已经不一样。outlink是第二个AVFilterLink，buffer也是做了变换的新的buffer
    return ff_filter_frame(outlink, out);  
}

通过ff_get_video_buffer，分配一个空buffer，该buffer用于存储变换的结果，
并会通过ff_filter_frame传递到下一个filter。 
do_conversion是一个真正做变换的函数，但其实如果要做的处理并不耗时，也不一定要用另一个线程来处理。
直接在该filter_frame做也行。 
处理好的新的数据，放在out，调用ff_filter_frame，传递给下一个filter。
注意，ff_filter_frame的oulink，对应上图的第二个AVFilterLink。

5.3.8. 再次走进ff_filter_frame // avfilter.c
如上已知，ff_filter_frame只做了一些基本检查，走到ff_filter_frame_framed。
故而我们直接看ff_filter_frame_framed


static int ff_filter_frame_framed(AVFilterLink *link, AVFrame *frame)
{
    //定义一个函数指针filter_frame。所指向的函数，参数为AVFilterLink *, AVFrame *,返回值为int
    int (*filter_frame)(AVFilterLink *, AVFrame *);

     //下一个AVFilterContext，对本例来说，就是系统默认的第三个滤镜，名字叫"format"，源码在vf_format.c
    AVFilterContext *dstctx = link->dst;
    AVFilterPad *dst = link->dstpad;
    AVFrame *out = NULL;
    int ret;

    if (!(filter_frame = dst->filter_frame))     //vf_format.c没有实现filter函数，因为返回为空
        filter_frame = default_filter_frame;    //所以函数会走到这，函数指针filter_frame 将指向default_filter_frame
    //省略300字

    ret = filter_frame(link, out);
    link->frame_count++;
    ff_update_link_current_pts(link, pts);
    return ret;
}

如注释所说，由于vf_format.c没有实现filter函数，所以此时的filter_frame指针，指向的是defalut_filter_frame。

5.3.9. default_filter_frame //avfilter.c
static int default_filter_frame(AVFilterLink *link, AVFrame *frame)
{
    //该函数没干啥，又调用ff_filter_frame了，第一个参数，换成第三个AVFilterLink了，第二个参数不变，frame默默的传递出去
    return ff_filter_frame(link->dst->outputs[0], frame);
}

此时link->dst->outputs[0]对应上图第三个AVFilterLink。

5.3.10. 第三次走进ff_filter_frame // avfilter.c
static int ff_filter_frame_framed(AVFilterLink *link, AVFrame *frame)
{
    //定义一个函数指针filter_frame。所指向的函数，参数为AVFilterLink *, AVFrame *,返回值为int
    int (*filter_frame)(AVFilterLink *, AVFrame *);

    //下一个AVFilterContext，对本例来说，就是系统默认的最后一个滤镜，名字叫"buffersink"，源码在bufffersink.c
    AVFilterContext *dstctx = link->dst;
    AVFilterPad *dst = link->dstpad;
    AVFrame *out = NULL;
    int ret;

    if (!(filter_frame = dst->filter_frame))   //指向buffersink.c实现的filte_frame函数
        filter_frame = default_filter_frame;
    //省略300字

    ret = filter_frame(link, out);
    link->frame_count++;
    ff_update_link_current_pts(link, pts);
    return ret;
}

此时的filter_frame指针，指向buffersink.c实现的filter_frame函数

5.3.11. filter_frame //buffersink.c

static int filter_frame(AVFilterLink *link, AVFrame *frame)
{
    AVFilterContext *ctx = link->dst;
    BufferSinkContext *buf = link->dst->priv;
    int ret;

    if ((ret = add_buffer_ref(ctx, frame)) < 0)
        return ret;
    //省略300字
    return 0;
}

static int add_buffer_ref(AVFilterContext *ctx, AVFrame *ref)
{
    BufferSinkContext *buf = ctx->priv;

    /* cache frame */
    //把buffer存到FIFO
    av_fifo_generic_write(buf->fifo, &ref, FIFO_INIT_ELEMENT_SIZE, NULL);
    return 0;
}

很清晰的看到，其实就是把buffer存到FIFO。
至此，把filter_frame的来龙去脉搞清楚啦

5.4 filter之后，ffmpeg如何编码

当我们写了一个filter，把视频做处理后，ffmpeg是如何把它编码的呢？

通过研究，发现编码的源头函数是reap_filters(…)，它会被transcode_step(…)函数调用。

5.4.1. reap_filters //ffmpeg.c
static int reap_filters(int flush)
{
    AVFrame *filtered_frame = NULL;//该指针将存储一个经过滤镜处理后的buffer，并送给encoder
    int i;

    /* Reap all buffers present in the buffer sinks */
    for (i = 0; i < nb_output_streams; i++) {//一路video，一路audio，那么nb_output_streams = 2
        OutputStream *ost = output_streams[i];
        OutputFile    *of = output_files[ost->file_index];
        AVFilterContext *filter;
        AVCodecContext *enc = ost->enc_ctx;
        int ret = 0;

        if (!ost->filter)
            continue;
        filter = ost->filter->filter;//OutputStream的filter指针指向buffersink.c定义的AVFilterContext。也就是本文讨论的，最后一个AVFilterContext

        if (!ost->filtered_frame && !(ost->filtered_frame = av_frame_alloc())) {
            return AVERROR(ENOMEM);
        }
        filtered_frame = ost->filtered_frame;

        while (1) {
            double float_pts = AV_NOPTS_VALUE; // this is identical to filtered_frame.pts but with higher precision
            //av_buffersink_get_frame_flags定义在buffersink.c，用于从FIFO读出一帧
            ret = av_buffersink_get_frame_flags(filter, filtered_frame,
                                               AV_BUFFERSINK_FLAG_NO_REQUEST);
            if (ret < 0) {
                //省略，检查ret
                //如果ret<0，不是别的错误，那认为还没有数据，跳出循环
                break;
            }
            switch (filter->inputs[0]->type) {
            case AVMEDIA_TYPE_VIDEO:
                //do_video_out函数将会做video编码
                do_video_out(of->ctx, ost, filtered_frame, float_pts);
                break;
            case AVMEDIA_TYPE_AUDIO:
                //do_audio_out函数将会做audioo编码
                do_audio_out(of->ctx, ost, filtered_frame);
                break;
            default:
                // TODO support subtitle filters
                av_assert0(0);
            }

            av_frame_unref(filtered_frame);
        }
    }

    return 0;
}

前一节说了，filter_frame(…)的最终结果是，把buffer存在了buffersink.c的FIFO里。 
那么，这一节，说的其实就是一个从buffersink的FIFO读数据，并编码的过程。 
从上面可知，av_buffersink_get_frame_flags函数，从buffersink读取一帧数据，放到filtered_frame。

5.4.2. do_video_out //ffmpeg.c
static void do_video_out(AVFormatContext *s,
                         OutputStream *ost,
                         AVFrame *next_picture,
                         double sync_ipts)
{
    int ret;
    AVCodecContext *enc = ost->enc_ctx;
    int nb_frames, nb0_frames, i;
    //省略300字
    for (i = 0; i < nb_frames; i++) {
        AVFrame *in_picture;
        if (i < nb0_frames && ost->last_frame) {
            in_picture = ost->last_frame;
        } else
            in_picture = next_picture;
        //省略300字
        ost->frames_encoded++;
        //开始编码
        ret = avcodec_encode_video2(enc, &pkt, in_picture, &got_packet);
    }
    ///省略300字
}

该函数很长，做了很多杂事，但关键代码就是调用编码函数avcodec_encode_video2

5.5 函数流程图

下面给出ffmpeg使用filter时的函数流程图，主要把和filter相关的函数拉出来！

图7： ffmpeg带filter的流程图

ffmpeg常规的avcodec_register_all(…), avfilter_register_all(…), av_register_all(…)等函数用于注册各种资源

transcode_init() : 主要用于初始化前文提到的各种结构体。

transcode_step : 主要工作： 解码->送filter过滤->编码->继续解码….

choose_output() : 用于选择一个OutputStream。比如有一个audio，一个video，那要根据pts策略，比如谁的pts比较小，就挑哪个OutputStream先干活。

transcode_frome_filter() : 用于选个一个InputStream，用于下一步的process_input()。

process_input() : 主要是解码，并把解码的buffer送往filter处理。

reap_filters() : 主要是，从filter的FIFO拿出buffer，并编码。

6. 参考资料

FFmpeg官网： http://www.ffmpeg.org

FFmpeg doc : http://www.ffmpeg.org/documentation.html

FFmpeg wiki : https://trac.ffmpeg.org/wiki

CSDN大牛：http://blog.csdn.net/leixiaohua1020/

https://blog.csdn.net/newchenxf/article/details/51364105

https://blog.csdn.net/fireroll/article/details/85319002