标题中bug：在最后按照离线包处说明，给出解决方式
公司需要使用最新版logstash来开发，我们自己开放性框架
- github下载对应源码：Logstash 7.10.1 github 地址
- 解压windows 直接解压/linux unzip xxxx.zip
- 生成java插件所需api 核心 jar 包文件：进入项目根目录执行-windows:gradlew.bat assemble linux：./gradlew assemble
- java 示列插件git地址：logstash-input-java_input_example
- plugin demo 根目录，增加java.properties 配置文件 gradle.properties，添加值：LOGSTASH_CORE_PATH=/data/logstashcpl/logstash-7.10.1/logstash-core 该核心路径指向上一步中核心jar包路径
- 打包插件：./gradlew gem
- 在引入 Logstash 官网下载地址 下载了 logstash-7.10.1
- 解压即可使用，进入logstash根目录
- 安装上面打包的插件：bin/logstash-plugin install --no-verify --local 你打包出来的插件
bin/logstash-plugin install --no-verify --local /data/plugins/logstash-input-java_input_example-master/logstash-input-java_input_example-1.0.1.gem
- 运行该插件：
bin/logstash -e "input {java_input_example {}} output {stdout { codec => rubydebug }}" - 查看该插件在logstash中的名称
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin list Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/ WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK .... logstash-input-imap logstash-input-java_input_example logstash-input-jm ..... #logstash-input-java_input_example 即为插件名
- 导出该插件用于离线环境安装（我司内网，没办法在内网编译安装）bin/logstash-plugin prepare-offline-pack --overwrite --output [包名] 插件名，上一步获取的插件名即可
#打离线包 [root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/java_input.zip logstash-input-java_input_example Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/ WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK Offline package created at: /data/java_input.zip You can install it with this command `bin/logstash-plugin install file:///data/java_input.zip`
- 安装该离线包：bin/logstash-plugin install file:///data/java_input.zip
我在执行上述步骤是遇到问题如下：
- demo 插件问题：该错误来源：
No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency $TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values:
dependsOn [downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles] 修改如下
dependsOn([downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles])
- demo插件问题：
/data/plugins/logstash-input-java_input_example-master/src/main/java/org/logstashplugins/JavaInputExample.java:3: error: package co.elastic.logstash.api does not exist import co.elastic.logstash.api.Configuration;
该错误原因为build.gradle 去匹配核心jar包时未匹配到，因为demo中
implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.?.?.jar") #logstash-core-?.?.?.jar 中 ? 匹配一个字符，一旦 x.y.z.jar包中版本超过9，达到两个字符就会匹配不到 #修改为： implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.??.?.jar") #或指定你生成的jar包版本
- 安装离线包：
ERROR: Something went wrong when installing file:///data//plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]
该错误是由于默认命名插件时以 java开头导致：build.gradle 文件中
pluginInfo.pluginType = "input" pluginInfo.pluginClass = "JavaInputExample" pluginInfo.pluginName = "java_input_example" //此处命名java开头导致
分析安装离线包导致的该问题：
根目录搜索该报错信息：
[root@VM_centos logstash-7.10.1]# grep Illformed -nr ./*
./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:110: raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
[root@VM_63_147_centos logstash-7.10.1]# vim ./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb +110
输出堆栈信息：puts caller
def self.parse(obj)
return ["=", obj] if Gem::Version === obj
unless PATTERN =~ obj.to_s
puts caller
raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
end
if $1 == ">=" && $2 == "0"
DefaultRequirement
else
[$1 || "=", Gem::Version.new($2)]
end
end
堆栈信息如下
Installing file: /data/plugins/java-input.zip /data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `block in initialize' /data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `map!' /data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `initialize' /data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:65:in `create' /data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/dependency.rb:60:in `initialize' /data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/dependency.rb:82:in `initialize' /data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:65:in `dependency' /data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `collect' /data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `inject!' /data/logstash/logstash-7.10.1/lib/pluginmanager/pack_installer/local.rb:58:in `execute' /data/logstash/logstash-7.10.1/lib/pluginmanager/install.rb:47:in `execute' /data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run' /data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11:in `execute' /data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run' /data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:133:in `run' /data/logstash/logstash-7.10.1/lib/pluginmanager/main.rb:64:in `<main>' ERROR: Something went wrong when installing file:///data/plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]
通过上述信息定位到：
vim ./lib/pluginmanager/pack_installer/pack.rb +65
class Pack
class GemInformation
EXTENSION = ".gem"
SPLIT_CHAR = "-"
JAVA_PLATFORM_RE = /-java/
DEPENDENCIES_DIR_RE = /dependencies/
attr_reader :file, :name, :version, :platform
def initialize(gem)
@file = gem
extracts_information
end
def dependency?
@dependency
end
def plugin?
!dependency?
end
private
def extracts_information
basename = ::File.basename(file, EXTENSION)
parts = basename.split(SPLIT_CHAR)
@dependency = ::File.dirname(file) =~ DEPENDENCIES_DIR_RE
puts "basename: #{basename}"
puts "parts: #{parts}"
if basename.match(JAVA_PLATFORM_RE)
@platform = parts.pop
@version = parts.pop
@name = parts.join(SPLIT_CHAR)
else
@platform = nil
@version = parts.pop
@name = parts.join(SPLIT_CHAR)
end
end
end
输出：basename、parts 可以分析出，因为打包名为 logstash-input-java_input_example-1.0.1 而 logstash 安装源码中，使用了
JAVA_PLATFORM_RE = /-java/
if basename.match(JAVA_PLATFORM_RE) 此处匹配到了，/-java/ 认为打包平台为 java 不是 ruby 导致
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin install file:///data/plugins/java-input.zip ... arguments: ["install", "file:///data/plugins/java-input.zip"] subcmd_class: LogStash::PluginManager::Install arguments: ["file:///data/plugins/java-input.zip"] Installing file: /data/plugins/java-input.zip basename: logstash-input-java_input_example-1.0.1
解决方案就比较多了，当然不建议修改源码，
直接修改插件名，别用 java 开头就好了
多说一点。
楼主排查的时候怀疑是不是打包就有问题，少打了 -java 参数到包名中去，捋了好久，具体过程就不说了，看结果吧
使用 bin/logstash-plugin prepare-offline-pack 去打包的话
[root@VM_centos logstash-7.10.1]# grep -rn "my collect:" ./
./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb:32: puts "my collect: #{collect_required_gems}"
[root@VM_centos logstash-7.10.1]# vim ./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb +32
module Paquet
class Gem
RUBYGEMS_URI = "https://rubygems.org/downloads"
attr_reader :gems, :ignores
def initialize(target_path, cache = nil)
@target_path = target_path
@gems = []
@ignores = []
@cache = cache
end
def add(name)
@gems << name
end
def ignore(name)
@ignores << name
end
def pack
Paquet::ui.info("Cleaning existing target path: #{@target_path}")
FileUtils.rm_rf(@target_path)
FileUtils.mkdir_p(@target_path)
puts "sq collect: #{collect_required_gems}"
package_gems(collect_required_gems)
end
输出打包命令去收集所需gems时参数都为：
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/plugins/java-input.zip logstash-input-java_input_example Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/ WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK arguments: ["prepare-offline-pack", "--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"] subcmd_class: LogStash::PluginManager::PrepareOfflinePack arguments: ["--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"] my test: ["logstash-input-java_input_example"] my collect: [#<Paquet::Dependency:0x51d76ad3 @version=#<Gem::Version "1.0.1">, @name="logstash-input-java_input_example", @platform="ruby">]
可以发现上面最后一行 collect 收集参数中 platform 就是ruby 不是 java
至此楼主也懒得研究了，研究这几天炸了。
反正 logstash 官网这插件 demo 质量太差，然后处理打包上，源码感觉也不够严谨，导致了这样的bug，分割字符串，正则匹配太SX了
最后楼主将修改后打出的包放到内网成功如下：
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash-plugin install file:///data/java-input.zip
Using JAVA_HOME defined java: /data/share/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Installing file: /data/java-input.zip
Install successful
SIS3.0.45.0 /home/fantom/party/logstash-7.10.1 #
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash -e "input {input_example_java {}} output {stdout { codec => rubydebug }}"
Using JAVA_HOME defined java: /data/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Sending Logstash logs to /data/logstash-7.10.1/logs which is now configured via log4j2.properties
[2020-12-21T21:14:01,451][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +indy +jit [linux-x86_64]"}
[2020-12-21T21:14:01,858][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/data/logstash-7.10.1/data/queue"}
[2020-12-21T21:14:01,878][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/fantom/party/logstash-7.10.1/data/dead_letter_queue"}
[2020-12-21T21:14:02,485][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-21T21:14:02,529][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"a33c9423-70fa-4794-9995-f02bc72e2192", :path=>"/home/fantom/party/logstash-7.10.1/data/uuid"}
[2020-12-21T21:14:05,000][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-21T21:14:08,510][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x21cc9c5e run>"}
[2020-12-21T21:14:11,199][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>2.66}
[2020-12-21T21:14:11,252][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-21T21:14:11,432][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
"@timestamp" => 2020-12-21T13:14:11.308Z,
"message" => "message 1 of 3 ",
"@version" => "1"
}
{
"@timestamp" => 2020-12-21T13:14:11.319Z,
"message" => "message 2 of 3 ",
"@version" => "1"
}
{
"@timestamp" => 2020-12-21T13:14:11.319Z,
"message" => "message 3 of 3 ",
"@version" => "1"
}
[2020-12-21T21:14:12,511][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2020-12-21T21:14:13,048][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2020-12-21T21:14:13,060][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-21T21:14:13,450][INFO ][logstash.runner ] Logstash shut down.