运维监控系统之Open-Falcon

一、Open-Falcon介绍

1、监控系统，可以从运营级别（基本配置即可），以及应用级别（二次开发，通过端口进行日志上报），对服务器、操作系统、中间件、应用进行全面的监控，及报警，对我们的系统正常运行的作用非常重要。

2、基础监控

CPU、Load、内存、磁盘、IO、网络相关、内核参数、ss 统计输出、端口采集、核心服务的进程存活信息采集、关键业务进程资源消耗、NTP offset采集、DNS解析采集，这些指标，都是open-falcon的agent组件直接支持的。

Linux运维基础采集项：http://book.open-falcon.org/zh/faq/linux-metrics.html

对于这些基础监控选项全部理解透彻的时刻，也就是对Linux运行原理及命令进阶的时刻。

3、第三方监控

术业有专攻，运行在OS上的应用甚多，Open-Falcon的开发团队不可能把所有的第三方应用的监控全部做完，这个就需要开源社区提供更多的插件，当前对于很多常用的第三方应用都有相关插件了。

4、JVM监控

对于Java作为主要开发语言的大多数公司，对于JVM的监控不可或缺。

每个JVM应用的参数，比如GC、类加载、JVM内存、进程、线程，都可以上报给Falcon，而这些参数的获得，都可以通过MxBeans实现。

使用 Java 平台管理 bean：http://www.ibm.com/developerworks/cn/java/j-mxbeans/

5、业务应用监控

对于业务需要监控的接口，比如响应时间等。可以根据业务的需要，上报相关数据到Falcon，并通过Falcon查看结果。

二、Zabbix、Nagios、Open-Falcon这3大开源运维监控工具的比较

三、Open-Falcon编写的整个脑洞历程

参考：http://mp.weixin.qq.com/s?__biz=MjM5OTcxMzE0MQ==&mid=400225178&idx=1&sn=c98609a9b66f84549e41cd421b4df74d

小米开源监控系统OpenFalcon应对高并发7种手段：http://h2ex.com/894

官网：

http://open-falcon.org/

https://book.open-falcon.org

https://github.com/open-falcon/falcon-plus/releases

说明：Open-Falcon部署分为前端（客户端）和后端（服务端）

四、后端部署（服务端）

参考官方文档：https://book.open-falcon.org/zh_0_2/quick_install/prepare.html

1、系统环境

[root@open-falcon-server ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

2、系统优化

#安装下载软件
yum install wget -y

#更换aliyun源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

#下载epel源
yum install epel-release.noarch -y
rpm -Uvh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
yum clean all
yum makecache

#下载常用软件
yum install git telnet net-tools tree nmap sysstat lrzsz dos2unix tcpdump ntpdate -y

#配置时间同步
ntpdate cn.pool.ntp.org

#更改主机名
hostnamectl set-hostname open-falcon-server
hostname open-falcon-server

#开启缓存
sed -i 's#keepcache=0#keepcache=1#g' /etc/yum.conf 
grep keepcache /etc/yum.conf

#关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

#关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service

3、软件环境准备

#安装 redis

yum install redis -y

#redis常用命令

redis-server     redis 服务端

redis-cli　　　　redis 命令行客户端

redis-benchmark  redis 性能测试工具

redis-check-aof 　　AOF文件修复工具

redis-check-dump  RDB文件修复工具

redis-sentinel 　　  Sentinel 服务端

#启动redis

#启动redis
[root@open-falcon-server ~]# redis-server &
[1] 3308
[root@open-falcon-server ~]# 3308:C 27 Oct 08:37:26.435 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
3308:M 27 Oct 08:37:26.436 * Increased maximum number of open files to 10032 (it was originally set to 1024).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.2.10 (00000000/0) 64 bit
  .-`` .-```.  ```/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 3308
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

3308:M 27 Oct 08:37:26.437 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3308:M 27 Oct 08:37:26.437 # Server started, Redis version 3.2.10
3308:M 27 Oct 08:37:26.437 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3308:M 27 Oct 08:37:26.438 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
3308:M 27 Oct 08:37:26.438 * The server is now ready to accept connections on port 6379

#安装mysql

[root@open-falcon-server ~]#wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm  

[root@open-falcon-server ~]# ll
total 12
-rw-------. 1 root root 1262 Oct 27  2017 anaconda-ks.cfg
-rw-r--r--. 1 root root 6140 Nov 12  2015 mysql-community-release-el7-5.noarch.rpm

[root@open-falcon-server ~]# rpm -ivh mysql-community-release-el7-5.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-release-el7-5    ################################# [100%]
[root@open-falcon-server ~]# ll /etc/yum.repos.d/
total 48
-rw-r--r--. 1 root root 2573 Nov 21  2014 CentOS-Base.repo
-rw-r--r--. 1 root root 1664 Dec  9  2015 CentOS-Base.repo.backup
-rw-r--r--. 1 root root 1309 Dec  9  2015 CentOS-CR.repo
-rw-r--r--. 1 root root  649 Dec  9  2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  290 Dec  9  2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root  630 Dec  9  2015 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Dec  9  2015 CentOS-Sources.repo
-rw-r--r--. 1 root root 1952 Dec  9  2015 CentOS-Vault.repo
-rw-r--r--. 1 root root  951 Jun 24 23:04 epel.repo
-rw-r--r--. 1 root root 1050 Jun 24 23:04 epel-testing.repo
-rw-r--r--. 1 root root 1209 Jan 29  2014 mysql-community.repo
-rw-r--r--. 1 root root 1060 Jan 29  2014 mysql-community-source.repo

[root@open-falcon-server ~]#yum install -y mysql-server

#启动mysql并设置开机自启动
[root@open-falcon-server ~]# systemctl start mysqld.service
[root@open-falcon-server ~]# systemctl enable mysqld.service

[root@open-falcon-server ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 2
Server version: 5.6.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> exit
Bye

#检查服务

[root@open-falcon-server ~]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      3308/redis-server * 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1443/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1634/master         
tcp6       0      0 :::3306                 :::*                    LISTEN      3283/mysqld         
tcp6       0      0 :::6379                 :::*                    LISTEN      3308/redis-server * 
tcp6       0      0 :::22                   :::*                    LISTEN      1443/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1634/master         
udp        0      0 0.0.0.0:68              0.0.0.0:*                           2042/dhclient       
udp        0      0 0.0.0.0:4817            0.0.0.0:*                           2042/dhclient       
udp6       0      0 :::20380                :::*                                2042/dhclient

4、初始化MySQL表结构（备注：导入表结构后，注意检查）

cd /tmp/ && git clone https://github.com/open-falcon/falcon-plus.git 
cd /tmp/falcon-plus/scripts/mysql/db_schema/
mysql -h 127.0.0.1 -u root -p < 1_uic-db-schema.sql
mysql -h 127.0.0.1 -u root -p < 2_portal-db-schema.sql
mysql -h 127.0.0.1 -u root -p < 3_dashboard-db-schema.sql
mysql -h 127.0.0.1 -u root -p < 4_graph-db-schema.sql
mysql -h 127.0.0.1 -u root -p < 5_alarms-db-schema.sql
rm -rf /tmp/falcon-plus/

#设置数据库密码 (备注：用户名：root ,密码: password )

[root@open-falcon-server open-falcon]# mysqladmin -uroot password "password"
Warning: Using a password on the command line interface can be insecure.

#检查导入的数据库表

[root@open-falcon-server open-falcon]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 35
Server version: 5.6.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| alarms             |
| dashboard          |
| falcon_portal      |
| graph              |
| mysql              |
| performance_schema |
| uic                |
+--------------------+
8 rows in set (0.00 sec)

mysql> exit
Bye

5、配置Go语言开发环境

#安装go语言开发包（备注：安装需要epel源）

yum install golang -y

#检查版本

[root@open-falcon-server ~]# go version
 go version go1.8.3 linux/amd64

#查看go的安装路径

[root@open-falcon-server ~]# find / -name go
/etc/alternatives/go
/var/lib/alternatives/go
/usr/bin/go
/usr/lib/golang/src/cmd/go  #需要的是这个路径
/usr/lib/golang/src/go
/usr/lib/golang/bin/go
/usr/lib/golang/pkg/linux_amd64/go

#设置环境变量GOROOT和GOPATH

export GOROOT=/usr/lib/golang
export GOPATH=/home

6、下载编译好的二进制版本

https://github.com/open-falcon/falcon-plus/releases

直接下载官方编译好的二进制版本(x86 64位平台)。

#开始安装open-falcon

[root@open-falcon-server ~]# export FALCON_HOME=/home/work
[root@open-falcon-server ~]# export WORKSPACE=$FALCON_HOME/open-falcon
[root@open-falcon-server ~]# mkdir -p $WORKSPACE

#上传软件
[root@open-falcon-server ~]# ll
-rw-r--r--. 1 root root 40031472 Oct 29 2017 open-falcon-v0.2.1.tar.gz

#解压
[root@open-falcon-server ~]# tar -xf open-falcon-v0.2.1.tar.gz -C $WORKSPACE
[root@open-falcon-server ~]# cd $WORKSPACE

#查看解压结果
[root@open-falcon-server open-falcon]# ll
total 3896
drwxrwxr-x. 7 501 501      67 Aug 15 19:16 agent
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 aggregator
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 alarm
drwxrwxr-x. 6 501 501      51 Aug 15 19:16 api
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 gateway
drwxrwxr-x. 6 501 501      51 Aug 15 19:16 graph
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 hbs
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 judge
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 nodata
-rwxrwxr-x. 1 501 501 3987469 Aug 15 19:16 open-falcon
lrwxrwxrwx. 1 501 501      16 Aug 15 19:16 plugins -> ./agent/plugins/
lrwxrwxrwx. 1 501 501      15 Aug 15 19:16 public -> ./agent/public/
drwxrwxr-x. 5 501 501      40 Aug 15 19:16 transfer

7、修改配置文件cfg.json

坑：

部分模块依赖连接数据库，因为如果不修改配置文件，aggregator模块会出现无法启动，graph、hbs、nodata、api、alarm模块会出现开启不报错但是状态为开启失败的情况。

如果需要每个模块都能正常启动，需要将上面模块的cfg.json的数据库信息进行修改。根据本教程的配置，需要修改配置文件所在的目录：

模块	配置文件所在路径
aggregator	/home/work/aggregator/config/cfg.json
graph	/home/work/graph/config/cfg.json
hbs	/home/work/hbs/config/cfg.json
nodata	/home/work/nodata/config/cfg.json
api	/home/work/api/config/cfg.json
alarm	/home/work/alarm/config/cfg.json

1）修改aggregator的配置文件

vim /home/work/aggregator/config/cfg.json