• 蓝鲸:安装SaaS组件bk_monitor失败分析解决


    使用./bk_install saas-o 安装发现bk_monitor(蓝鲸监控)组件报错“ERROR deploy failed: timeout”。

    单独尝试安装各个组件:

    #故障自愈
    [root@rbtnode1 install]# ./bk_install saas-o bk_fta_solutions
    
    #日志检索
    [root@rbtnode1 install]# ./bk_install saas-o bk_log_search
    
    #节点管理
    [root@rbtnode1 install]# ./bk_install saas-o bk_nodeman
    
    #标准运维
    [root@rbtnode1 install]# ./bk_install saas-o bk_sops
    
    #蓝鲸监控
    [root@rbtnode1 install]# ./bk_install saas-o bk_monitor
    

    发现前面几个bk_fta_solutions、bk_log_search、bk_nodeman、bk_sops都可以安装成功,唯独对bk_monitor安装,依然报错如下:

    [root@rbtnode1 install]# ./bk_install saas-o bk_monitor
    省略输出..
    2020-03-09 13:27:36 125  INFO    check deploy result. retry 132
    2020-03-09 13:27:39 125  INFO    check deploy result. retry 133
    2020-03-09 13:27:39 134  ERROR  deploy failed: timeout
    [192.168.1.6]20200309-132739 153   Deploy saas bk_monitor failed.
    [192.168.1.6]20200309-132739 47   Abort
    

    进一步查看agent日志(/data/bkce/logs/paas_agent/agent.log),最终因为部署任务timeout而终止,未见其他明显报错:

    2020/03/09 13:24:57 job.go:279: Building wheels for collected packages: gevent, netifaces, arrow, msgpack-python, wrapt, itypes, backports.shutil-get-terminal-size, simplegeneric, scandir
    
    2020/03/09 13:24:57 job.go:279:   Running setup.py bdist_wheel for gevent: started
    
    2020/03/09 13:27:32 job.go:279:   Running setup.py bdist_wheel for gevent: still running...
    
    2020/03/09 13:27:38 job.go:297: Deployment task execution timeout
    

    查了些网上资料,说是因为机器配置不够,增加核数为6即可解决,但实际我测试无效,报错不变;
    在蓝鲸官方群咨询,客服给出一个解决方案:

    但是实际这个Case和我这里遇到的情况并不一样,因为我这没有看到那个error。
    晚上重新整理下思路,借鉴案例中清理环境的方式,然后重新部署,这次agent.log看到报错信息了:

    2020/03/10 02:29:54 job.go:279:   File "/opt/py27_e/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
    
    2020/03/10 02:29:54 job.go:279:     packet.check_error()
    
    2020/03/10 02:29:54 job.go:279:   File "/opt/py27_e/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
    
    2020/03/10 02:29:54 job.go:279:     err.raise_mysql_exception(self._data)
    
    2020/03/10 02:29:54 job.go:279:   File "/opt/py27_e/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
    
    2020/03/10 02:29:54 job.go:279:     _check_mysql_exception(errinfo)
    
    2020/03/10 02:29:54 job.go:279:   File "/opt/py27_e/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
    
    2020/03/10 02:29:54 job.go:279:     raise InternalError(errno, errorvalue)
    
    2020/03/10 02:29:54 job.go:279: django.db.utils.InternalError: (1049, u"Unknown database 'bkdata_monitor_alert'")
    
    2020/03/10 02:29:55 job.go:304: error waiting for Cmd exit status 1
    

    这提示居然是没有这个名称为bkdata_monitor_alert的数据库??
    结合之前的agent日志是确认有建表操作成功的,说明是环境清理操作很可能把对应组件的库也给删除了。

    这里先不深究,直接查看当前的数据库列表:

    MySQL [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | bk_fta_solutions   |
    | bk_log_search      |
    | bk_monitor         |
    | bk_nodeman         |
    | bk_sops            |
    | bksuite_common     |
    | job                |
    | jobLog             |
    | mysql              |
    | open_paas          |
    | performance_schema |
    | sys                |
    +--------------------+
    13 rows in set (0.00 sec)
    

    果然没有这个bkdata_monitor_alert库,这里先直接尝试创建一个空库试下:

    MySQL [(none)]> create database bkdata_monitor_alert character set utf8;
    Query OK, 1 row affected (0.01 sec)
    

    再次尝试bk_monitor的安装:

    # 再次安装bk_monitor
    [root@rbtnode1 install]# ./bk_install saas-o bk_monitor
    
    # 监控agent.log
    [root@rbtnode1 paas_agent]# pwd
    /data/bkce/logs/paas_agent
    [root@rbtnode1 paas_agent]# tail -20f agent.log 
    

    发现这次agent.log日志最终显示Job正常完成了:

    省略部分日志..
    
    2020/03/10 02:45:38 job.go:279:   Applying sessions.0001_initial... OK
    
    2020/03/10 02:45:38 job.go:279: ------change db success------
    
    2020/03/10 02:47:25 job.go:279: ------ start app server ------
    
    2020/03/10 02:47:25 job.go:279: su: ignore --preserve-environment, it's mutually exclusive to --login.
    
    2020/03/10 02:47:25 job.go:279: /etc/profile: line 77: ulimit: open files: cannot modify limit: Operation not permitted
    
    2020/03/10 02:47:25 job.go:279: /etc/profile: line 78: ulimit: open files: cannot modify limit: Operation not permitted
    
    2020/03/10 02:47:25 job.go:279: /etc/profile: line 79: ulimit: open files: cannot modify limit: Operation not permitted
    
    2020/03/10 02:47:25 job.go:279: /etc/profile: line 80: ulimit: open files: cannot modify limit: Operation not permitted
    
    2020/03/10 02:47:26 job.go:279: Last login: Mon Mar  9 14:01:54 CST 2020
    
    2020/03/10 02:47:28 job.go:279: Job Done
    
    2020/03/10 02:47:28 job.go:306: RunJob end ... ...
    

    赶紧去看下安装的窗口,发现这次bk_monitor终于安装成功了:

    [root@rbtnode1 install]# ./bk_install saas-o bk_monitor
    省略部分日志..
    
    2020-03-10 02:47:24 125  INFO    check deploy result. retry 107
    2020-03-10 02:47:26 125  INFO    check deploy result. retry 108
    2020-03-10 02:47:29 125  INFO    check deploy result. retry 109
    2020-03-10 02:47:30 131  INFO   bk_monitor have been deployed successfully
    [192.168.1.6]20200310-024730 151   SaaS application bk_monitor has been deployed successfully
    [192.168.1.6]20200310-024730 56   install saas-o(bk_monitor) done
    

    登陆蓝鲸的工作台,也确认这次蓝鲸监控组件已经安装成功,可以正常操作了。

  • 相关阅读:
    influxdb时序数据库之随想其他列式存储数据库
    ss 命令参数 redis-benchmark 以及POSTMAN, CURL分析网络
    HTTP2.0
    我的2020工作总结
    2021-03-31 JDK8 字符串常量池
    try catch与spring的事务回滚
    Spring主动触发事务回滚
    Redis五种数据类型及应用场景
    java面试题
    ubuntu linux mysql 安装 基本操作 命令
  • 原文地址:https://www.cnblogs.com/jyzhao/p/12453294.html
Copyright © 2020-2023  润新知