• flume:spooldir采集日志,kafka输出的配置问题


    flume配置:

    #DBFile
    DBFile.sources = sources1  
    DBFile.sinks = sinks1  
    DBFile.channels = channels1  
      
    # DBFile-DB-Source 
    DBFile.sources.sources1.type = spooldir
    DBFile.sources.sources1.spoolDir =/var/log/apache/flumeSpool//db
    DBFile.sources.sources1.inputCharset=utf-8
      
    # DBFile-Sink  
    DBFile.sinks.sinks1.type = org.apache.flume.sink.kafka.KafkaSink  
    DBFile.sinks.sinks1.topic = DBFile
    DBFile.sinks.sinks1.brokerList = hdp01:6667,hdp02:6667,hdp07:6667
    DBFile.sinks.sinks1.requiredAcks = 1  
    DBFile.sinks.sinks1.batchSize = 2000  
      
    # DBFile-Channel
    DBFile.channels.channels1.type = memory
    DBFile.channels.channels1.capacity = 10000
    DBFile.channels.channels1.transactionCapacity = 1000
    
    # DBFile-Source And Sink to the channel
    DBFile.sources.sources1.channels = channels1
    DBFile.sinks.sinks1.channel = channels1

           故障现象:第一次上传文件时,flume能很快处理文件,后面上传还是显示文件未处理。如果重启flume服务,又能立刻处理。

    经测试,问题的原因在这个配置上:DBFile.sinks.sinks1.requiredAcks = -1

          requiredAcks 的官方解释:How many replicas must acknowledge a message before its considered successfully written. Accepted values are 0 (Never wait for acknowledgement),

    1 (wait for leader only), -1 (wait for all replicas) Set this to -1 to avoid data loss in some cases of leader failure.

        把这个值改为1就好了。

  • 相关阅读:
    zz:android sharedpreferences用法
    Android viewHolder
    Python学习笔记~
    zz:Java多线程编程总结
    zz:Android 2.1 源码结构
    onSaveInstanceState的用法
    zz:如何学习Linux操作系统
    Python学习笔记II
    Python类型转换
    zz:Android应用程序基础
  • 原文地址:https://www.cnblogs.com/huiy/p/6128029.html
Copyright © 2020-2023  润新知