在清洗日志时，有一个应用场景，就是TCP输出时，须要在一个主机挂了的情况下，自已切换到下一个可用入口。而原tcp output仅支持单个目标主机设定。故本人在原tcp的基础上，开发出tcp_multihost输出插件，来满足此场景。
插件在一開始的时候会随机选择一个链路，而在链路出错连续超过３(默认)次后会尝试数组中下一个主机
github: http://github.com/xiaohelong2005
Logstash版本号:1.4.2
文件位置:
# encoding: utf-8
require "logstash/outputs/base"
require "logstash/namespace"
require "thread"
#@auhor:xiaohelong
#@date:2014-10-24
#@email:xiaohelong2005@gmail.com
# Write events over a TCP socket.
#Auto select the host from iplist to tolerate the link
# Each event json is separated by a newline.
#
# Can connect to a server,
class LogStash::Outputs::Tcp_Multihost < LogStash::Outputs::Base
config_name "tcp_multihost"
milestone 0
default :codec, "json"
# the address to connect to.
#hosts config example
config :hosts, :validate => :array, :required => true,:default=>{}
config :reconnect_times, :validate => :number,:default=>3
# When connect failed,retry interval in sec.
config :reconnect_interval, :validate => :number, :default => 10
#last available host
@hosts_size=0
#last available host ip
@host="0.0.0.0"
#last available port
@port=9200
#retry action count,if retry_count<=0,we need to update the host and port , also include retry_count itself
@retry_count=0
#get the desgined index data
@initloc=0
public
def register
require "stud/try"
#here we use the recorded host,if recorded host is not available, we need update it
@retry_count=@reconnect_times
@hosts_size=@hosts.length
@logger.info("length:#@hosts_size; hosts:#@hosts")
@initloc=Random.rand(@hosts_size)#generate 0-(hosts_size-1) int
@logger.info("initloc:#@initloc")
icount=0;
@hosts.each do |hosthash|
@logger.info("hosthash info",hosthash)
end#do
@host=@hosts[@initloc].keys[0]
@port=@hosts[@initloc][@host]
client_socket = nil
@codec.on_event do |payload|
begin
@retry_count=@reconnect_times#here we need to init retry mark
client_socket = connect unless client_socket
r,w,e = IO.select([client_socket], [client_socket], [client_socket], nil)
# don't expect any reads, but a readable socket might
# mean the remote end closed, so read it and throw it away.
# we'll get an EOFError if it happens.
client_socket.sysread(16384) if r.any?
# Now send the payload
client_socket.syswrite(payload) if w.any?
@logger.info("tcp output info:", :host => @host, :port => @port,
:exception => e, :backtrace => e.backtrace)
rescue => e
@logger.warn("tcp output exception", :host => @host, :port => @port,
:exception => e, :backtrace => e.backtrace)
client_socket.close rescue nil
client_socket = nil
@retry_count-=1
@logger.info("retry_count:#@retry_count")
if @retry_count<=0
@initloc+=1
@initloc=@initloc%@hosts_size #update init location
@host=@hosts[@initloc].keys[0]
@port=@hosts[@initloc][@host]
@retry_count=@reconnect_times #update retry_count
@logger.info("retry_count <=0,initloc:#@initloc,retry_count=#@retry_count:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace)
end
sleep @reconnect_interval
retry
end
end
end # def register
private
def connect
Stud::try do
return TCPSocket.new(@host,@port)
end
end # def connect
public
def receive(event)
return unless output?(event)
@codec.encode(event)
end # def receive
end # class LogStash::Outputs::Tcp_multihost
配置说明(我放在LOGSTASH_HOME/config)：
output{
tcp_multihost{
hosts=>[
{"127.0.0.1"=>"9202"},
{"localhost"=>"9201"},
{"127.0.0.1"=>"9203"},
{"127.0.0.1"=>"9204"}
] #主机列表
workers =>16 #线程,默认1
reconnect_times=>3 # 默认3次, 尝试多少次数后切换
reconnect_interval=>3 #默认10秒，失败重连间隔
}}
调用运行:
"LOGSTASH_HOME/bin/logstash" agent --debug -f "LOGSTASH_HOME/config/shipper.config" --pluginpath "LOGSTASH_HOME"
NC接收端能够尝试:
nc -lkv 9201 之类的