• Flume Source官网剖析(博主推荐)


      

       不多说,直接上干货!

      一切来源于flume官网

    http://flume.apache.org/FlumeUserGuide.html

    Flume Sources

       

    Avro Source

       官网给的例子是

    a1.sources = r1
    a1.channels = c1
    a1.sources.r1.type = avro
    a1.sources.r1.channels = c1
    a1.sources.r1.bind = 0.0.0.0
    a1.sources.r1.port = 4141

     而我们常用的一般是,

    agent1.sources = avro-source1
    agent1.channels = ch1
    
    
    
    #Define and configure an Spool directory source
    agent1.sources.avro-source1.channels=ch1
    agent1.sources.avro-source1.type=avro
    agent1.sources.avro-source1.bind=0.0.0.0
    agent1.sources.avro-source1.port=4141


    a1.sources.r1.interceptors = i1
    a1.sources.r1.interceptors.i1.type = com.djt.flume.interceptor.BehaviorIterceptor$BehaviorIterceptorBuilder


     

       

    Thrift Source

      

    Exec Source

    JMS Source

    Spooling Directory Source(常用)

      官网上给的参考例子是

    a1.channels = ch-1
    a1.sources = src-1
    
    a1.sources.src-1.type = spooldir
    a1.sources.src-1.channels = ch-1
    a1.sources.src-1.spoolDir = /var/log/apache/flumeSpool
    a1.sources.src-1.fileHeader = true

      而我们常用的一般是,

    agent1.sources = spool-source1
    agent1.channels = ch1
    
    
    
    #Define and configure an Spool directory source
    agent1.sources.spool-source1.channels=ch1
    agent1.sources.spool-source1.type=spooldir
    agent1.sources.spool-source1.spoolDir=/home/hadoop/data/flume/sqooldir
    agent1.sources.spool-source1.ignorePattern=event(_d{4}-d{2}-d{2}\_d{2}\_d{2})?.log(.COMPLETED)?
    agent1.sources.spool-source1.deserializer.maxLineLength=10240

    Taildir Source

    Twitter 1% firehose Source

    Kafka Source(常用)

       官网给的例子是

    Example for topic subscription by comma-separated topic list.

    tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
    tier1.sources.source1.channels = channel1
    tier1.sources.source1.batchSize = 5000
    tier1.sources.source1.batchDurationMillis = 2000
    tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
    tier1.sources.source1.kafka.topics = test1, test2
    tier1.sources.source1.kafka.consumer.group.id = custom.g.id

     Example for topic subscription by regex

    tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
    tier1.sources.source1.channels = channel1
    tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
    tier1.sources.source1.kafka.topics.regex = ^topic[0-9]$
    # the default kafka.consumer.group.id=flume is used

      具体官网里,还给了Security and Kafka Source、TLS and Kafka Source、Kerberos and Kafka Source。自行去看吧

    NetCat Source

    Sequence Generator Source

    Syslog Sources

    Syslog TCP Source

    Multiport Syslog TCP Source

    Syslog UDP Source

    HTTP Source

       官网还提供了JSONHandler、BlobHandler。这里不多说,自行去看吧

    Stress Source

    Legacy Sources

    Avro Legacy Source

    Thrift Legacy Source

    Custom Source

    Scribe Source

       

     大家自己去看,罗列出来,是让大家,不要局限于博客本身,眼光要放宽和多看官网,一切来源于官网。

      对于大数据无论是各种开源项目,看官网是学习的最好办法,再加上他人的中文博客。不要觉得英文恐惧,专业英语也就那样!变成高手和大牛,加油吧,zhouls!

  • 相关阅读:
    hihoCoder #1077 RMQ问题再临-线段树
    ms sql 获取字符串首字母
    如何设置gen_server在退出时执行相关操作
    C++拾遗
    [置顶] Linux下文件和目录权限说明
    Android百度地图之显示地图
    USACO March. 2012
    JNI之HelloWorld
    复习C语言系列二:动态调用函数指针数组
    HDU2527:Safe Or Unsafe(哈弗曼树)
  • 原文地址:https://www.cnblogs.com/zlslch/p/6761352.html
Copyright © 2020-2023  润新知