Elasticsearch + Logstash + Kibana + SearchGuard 安装部署
环境说明
系统以及java版本
| 系统版本 | JAVA 版本 |
|---|---|
| CentOS 7.4.1708 | 1.8 |
Elasticsearch node 配置
三台服务器都添加host 文件
| IP | Elasticsearch node |
|---|---|
| 10.3.245.25 | node-25 |
| 10.3.245.40 | node-40 |
| 10.3.245.65 | node-65 |
ELK 版本信息
| ElasticSearch版本 | Kibana版本 | Logstash | SearchGuard版本 |
|---|---|---|---|
| 6.6.1 | 6.6.1 | 6.6.1 | 25.5 |
SearchGuard 插件版本
| ES对应的SearchGuard插件版本 | Kibana对应的SearchGuard插件版本 |
|---|---|
| 25.5 | 18.5 |
JAVA 安装
三台服务器都需要安装
java 版本是1.8
解压配置环境变量
# 解压
tar fzx jdk-8u181-linux-x64.tar.gz
# 移动到/usr/local
mv jdk1.8.0_181 /usr/local/
# 配置环境变量
# 将下面两行放到/etc/profile 文件最后
export JAVA_HOME=/usr/local/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH
# 加载使其生效
source /etc/profile
检查是否生效
# 检查是否生效
java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
Elasticsearch 安装
在安装elk 之前 java必须提前安装，三台服务器都需要安装
es 的版本是 6.6.1
基本配置
# 本次使用的是rpm包安装
yum localinstall elasticsearch-6.6.1.rpm -y
# 修改jvm 配置 将内存大小给4G，如果内存比较多，可以给8G，或者16G，本机内存比较小，所以给4G
vim /etc/elasticsearch/jvm.options
-Xms4g
-Xmx4g
# 修改es配置文件参数，下面以10.3.245.25 为例
grep -Ev "^$|^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: elk
node.name: node-25
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.3.245.25
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["node-25", "node-65","node-40"]
discovery.zen.minimum_master_nodes: 2
# restart Elasticsearch
/etc/init.d/elasticsearch restart
Search Guard 插件安装
Elasticsearch 版本对应的searchguard 插件版本，需要在searchguard 官网查看并下载
安装插件
# 切换到es bin 目录
cd /usr/share/elasticsearch/bin/
# 安装有两种方法：
#第一种是在线安装
./elasticsearch-plugin install com.floragunn:search-guard-6:6.6.1-25.5
#第二种是线下安装
./elasticsearch-plugin install -b file:///root/search-guard-6-6.6.1-25.5.zip
# 本次使用第二种线下安装
配置Elasticsearch
# 进行demo模式的安装
cd /usr/share/elasticsearch/plugins/search-guard-6/tools
ls
hash.bat hash.sh install_demo_configuration.sh sgadmin.bat sgadmin_demo.sh sgadmin.sh
chmod +x install_demo_configuration.sh
./install_demo_configuration.sh # 三个y 即可
# 安装完成之后，会在/etc/elasticsearch/elasticsearch.yml 配置文件中添加一些参数。
# 请把https 更新为http，修改参数searchguard.ssl.http.enabled 的值为false
# 下面是具体配置完的参数
grep -Ev "^$|^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: elk
node.name: node-25
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.3.245.25
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["node-25", "node-65","node-40"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: "Authorization"
searchguard.ssl.transport.pemcert_filepath: esnode.pem
searchguard.ssl.transport.pemkey_filepath: esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: false
searchguard.ssl.http.pemcert_filepath: esnode.pem
searchguard.ssl.http.pemkey_filepath: esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: ["sg_all_access"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
xpack.security.enabled: false
# 在es上配置还有一步
# 将下面的配置添加到 /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240
elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
# 重启elasticsearch 服务
/etc/init.d/elasticsearch restart
SearchGuard 多用户设置
目录位置
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig
SearchGuard 各个文件的说明
- sg_internal_users.yml: 存储用户名密码，密码可以使用plugin/tools/hash.sh生成
- sg_roles.yml：权限设置，定义什么类型的权限
- sg_roles_mapping.yml： 映射角色关系，可以把权限映射给用户，也可以映射给用户组
- sg_action_groups.yml： 定义一些用户动作的权限与es索引之间的关系
- sg_config.yml：全局设置
实例 （创建一个用户）
注： 如果是集群，则下面的操作在所有的服务器都要操作一遍
-
创建用户密码
# cd /usr/share/elasticsearch/plugins/search-guard-6/tools # ls hash.bat hash.sh install_demo_configuration.sh sgadmin.bat sgadmin_demo.sh sgadmin.sh # sh hash.sh -p oapassword $2y$12$d5adFlpwkVFfyL7awgSbPekVsi7v7vfrNFQWCH98/7Oh4dtCHH5Iy #编辑 sg_internal_users.yml 文件 #将下面几行代码放到最后的位置就可以 # password is: oapassword oauser: hash: $2y$12$d5adFlpwkVFfyL7awgSbPekVsi7v7vfrNFQWCH98/7Oh4dtCHH5Iy roles: - sg_oauser -
创建角色 sg_osuser
# vim sg_roles.yml # 将下面的代码放到最后就可以 sg_oauser: cluster: - UNLIMITED indices: 'kibana*': '*': - READ '?nkibana*': '*': - READ 'logstash-oa*': '*': - CRUD 'oa-*': '*': - CRUD -
给用户赋予角色
# vim sg_roles_mapping.yml # 将下面代码放到最后就可以 sg_oauser: readonly: true backendroles: - sg_oauser -
更新用户配置
# cd /usr/share/elasticsearch/plugins/search-guard-6/tools # ls hash.bat hash.sh install_demo_configuration.sh sgadmin.bat sgadmin_demo.sh sgadmin.sh # ./sgadmin.sh -cd ../sgconfig/ -icl -nhnv -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/kirk.pem -key /etc/elasticsearch/kirk-key.pem -h IP地址
权限说明
权限定义
权限配置
权限映射
检测Elasticsearch
在浏览器上输入http://10.3.245.25:9200 ，正常情况下会弹出一个密码验证框，输入用户名和密码,SearchGuard的默认用户名和密码都是admin
{
name: "node-25",
cluster_name: "elk",
cluster_uuid: "bzeCw1L9RGi1dlCqOXDC4A",
version: {
number: "6.6.1",
build_flavor: "default",
build_type: "rpm",
build_hash: "1fd8f69",
build_date: "2019-02-13T17:10:04.160291Z",
build_snapshot: false,
lucene_version: "7.6.0",
minimum_wire_compatibility_version: "5.6.0",
minimum_index_compatibility_version: "5.0.0"
},
tagline: "You Know, for Search"
}
故障原因
如果Elasticsearch 没有重启成功，请检查日志 /var/log/elasticsearch/elk.log (elk.log 其中elk是集群名称)
- 配置文件中的配置参数，是否有写错或者漏写
- 日志文件路径是否由于权限问题没有生成
- java 环境变量是否没有识别到，在/etc/sysconfig/elasticsearch 配置文件中配置JAVA_HOME
Kibana 安装
kibana 也需要依赖JAVA
Kibana 版本为 6.6.1
安装
yum localinstall kibana-6.6.1-x86_64.rpm -y
安装SearchGuard插件
# 切换到kibana bin 目录
cd /usr/share/kibana/bin/
ls
kibana kibana-keystore kibana-plugin
# 安装插件
./kibana-plugin install file:///root/search-guard-kibana-plugin-6.6.1-18.5.zip
配置
# kibana 配置文件
grep -Ev "^$|^#" /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.3.245.25"
server.name: "kibana-server"
elasticsearch.hosts: ["http://10.3.245.25:9200","http://10.3.245.40:9200","http://10.3.245.65:9200"]
kibana.index: ".kibana"
elasticsearch.username: "admin"
elasticsearch.password: "admin"
elasticsearch.pingTimeout: 1500
elasticsearch.requestTimeout: 30000
elasticsearch.requestHeadersWhitelist: [ "authorization","sgtenant" ]
elasticsearch.shardTimeout: 30000
elasticsearch.startupTimeout: 5000
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.watcher.enabled: false
xpack.security.enabled: false
#重启kibana
/etc/init.d/kibana restart
访问Kibana
输入用户名和密码 admin/admin 登录
Logstash 安装
软件下载安装
logstash 版本为 6.6.1
# yum 安装
yum localinstall logstash-6.6.1.rpm -y
# 安装后logstash 目录 /usr/share/logstash
下载logstash pattern 目录
配置文件
/etc/logstash/jvm.options 中的
-Xms4g
-Xmx4g
修改为4G 或更多
# 修改配置文件
cd /etc/logstash
ls /etc/logstash/
conf.d jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options
#启动命令(默认情况下logstash 不提供启停脚本)
/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d -r -l /tmp
# --path.settings 配置文件路径
# -f /etc/logstash/conf.d 清洗数据路径
# -r 如果有配置更新，会自动更新
# -l 日志目录
# logstash 说明
Usage:
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: "m8-ops-elk-02.test.xesv5.com")
-f, --path.config CONFIG_PATH Load the logstash config from a specific file
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string CONFIG_STRING Use the given string as the configuration
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and will be becoming more strict to better
handle illegal and ambbiguous inputs in a
future release of Logstash.
Available MODEs are:
- `LEGACY`: parse with the legacy parser,
which is known to handle ambiguous- and
illegal-syntax in surprising ways;
warnings will not be emitted.
- `COMPAT`: warn once for each distinct
ambiguous- or illegal-syntax input, but
continue to expand field references with
the legacy parser.
- `STRICT`: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
The MODE can also be set with
`config.field_reference.parser`
(default: "COMPAT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 4)
--java-execution Use Java execution engine.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "/usr/share/logstash/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "/usr/share/logstash/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: 3000000000)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: "/usr/share/logstash/config")
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
测试Logstash 输出到Elasticsearch
# cat /etc/logstash/conf.d/message.conf
input {
http {
port => 7474
}
}
filter {
grok {
patterns_dir => ['/etc/logstash/pattern']
pattern_definitions => {
"DATETIME" => "%{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}%{TIME}"
}
match => {
"message" => '%{DATETIME:datetime} %{HOSTNAME:hostname} (%{HOSTNAME:system}:|%{USERNAME:system}\[%{USERNAME}\]:) %{GREEDYDATA:message}'
}
overwrite => ["message"]
remove_field => ["headers"]
}
}
output {
elasticsearch {
hosts => ["10.3.245.65:9200","10.3.245.25:9200"]
index => "test"
user => admin
password => admin
}
}
启动Logstash
Logstash 启动的时候 增加 -t 是进行语法检测
# 启动logstash
/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d -r -l /tmp/
Sending Logstash logs to /tmp/ which is now configured via log4j2.properties
[2020-01-04T23:51:26,208][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-01-04T23:51:26,221][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2020-01-04T23:51:31,358][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-01-04T23:51:31,719][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://admin:xxxxxx@10.3.245.65:9200/, http://admin:xxxxxx@10.3.245.25:9200/]}}
[2020-01-04T23:51:31,903][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://admin:xxxxxx@10.3.245.65:9200/"}
[2020-01-04T23:51:31,943][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-01-04T23:51:31,946][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-01-04T23:51:31,951][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://admin:xxxxxx@10.3.245.25:9200/"}
[2020-01-04T23:51:31,979][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.3.245.65:9200", "//10.3.245.25:9200"]}
[2020-01-04T23:51:31,988][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2020-01-04T23:51:32,007][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-01-04T23:51:32,320][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7d814629 run>"}
[2020-01-04T23:51:32,332][INFO ][logstash.inputs.http ] Starting http input listener {:address=>"0.0.0.0:7474", :ssl=>"false"}
[2020-01-04T23:51:32,366][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-01-04T23:51:32,564][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
从postman 输出信息到logstash