• 配置systemctl(zookeeper、hadoop、redis)


    ==背景==

    给公司搭建了一个建议的大数据平台,组件包括zookeeper、hadoop、flink、influxdb、redis、mysql、rocketmq

    因为zookeeper、hadoop是平台的基础组件,打算配置systemctl守护进程,用来挂掉自动重启,以减少运维负担。

    ==规划==

    1、每台机器设置service文件的管理文件夹:/home/radmin/rexel-dev

    2、文件组成,为每个service生成两个文件:xxx.service及xxx.service.run。.run是.service需要执行的脚本,负责环境依赖

    3、软连接:在systemctl目录下建立软连接,连接到rexel-dev下的具体文件

    4、所有.run文件赋予777权限。

    ==准备==

    1、创建pid文件存放路径

    for i in {1..3}; do ssh vm$i mkdir -p /home/radmin/data/pids/hadoop; done

    2、创建service文件存放路径

    for i in {1..3}; do ssh vm$i mkdir -p /home/radmin/rexel-dev; done

    3、配置系统环境变量

    export HADOOP_PID_DIR=/home/radmin/data/pids/hadoop
    export YARN_PID_DIR=/home/radmin/data/pids/hadoop

    ==zookeeper==

    1、文件规划

    rexel-zookeeper.service
    rexel-zookeeper.service.run

    2、配置run文件

    文件:rexel-zookeeper.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/zookeeper-3.5.6/bin/zkServer.sh start

    3、配置service文件

    文件:rexel-zookeeper.service

    [Unit]
    Description=Zookeeper Service
    After=network.target
    ConditionPathExists=/home/radmin/zookeeper-3.5.6/conf/zoo.cfg
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/zk/dataDir/zookeeper_server.pid
    ExecStart=/home/radmin/rexel-dev/rexel-zookeeper.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    4、配置软连接

    命令:

    ln -s /home/radmin/rexel-dev/rexel-zookeeper.service /usr/lib/systemd/system/rexel-zookeeper.service

    5、加载生效

    命令:

    systemctl daemon-reload

    6、启动服务

    命令:

    systemctl status rexel-zookeeper.service

    【小贴士】

    任何增加、修改或删除systemctl,都需要执行systemctl daemon-reload命令,使修改生效。

    【小插曲】

    service启动失败,通过journalctl -xe命令查看日志,发现提示

    Can't open PID file /home/radmin/data/pid/zookeeper.pid (yet?) after start: No such file or directory

    发现原因是配置的pid文件路径不正确,将pid文件的路径和zoo.cfg中的dataDir路径一致,重新启动service之后,恢复正常。

    ==hadoop==

    1、文件规划

    rexel-hdfs-journalnode.service
    rexel-hdfs-journalnode.service.run
    rexel-hdfs-namenode.service
    rexel-hdfs-namenode.service.run
    rexel-hdfs-zkfc.service
    rexel-hdfs-zkfc.service.run
    rexel-hdfs-datanode.service
    rexel-hdfs-datanode.service.run
    rexel-yarn-nodemanager.service
    rexel-yarn-nodemanager.service.run
    rexel-yarn-resourcemanager.service
    rexel-yarn-resourcemanager.service.run

    2、配置run文件

    文件1:rexel-hdfs-journalnode.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/hadoop-daemon.sh start journalnode

    文件2:rexel-hdfs-namenode.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/hadoop-daemon.sh start namenode

    文件3:rexel-hdfs-zkfc.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/hadoop-daemon.sh start zkfc

    文件4:rexel-hdfs-datanode.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/hadoop-daemon.sh start datanode

    文件5:rexel-yarn-resourcemanager.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/yarn-daemon.sh start resourcemanager

    文件6:rexel-yarn-nodemanager.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/hadoop-2.8.3/sbin/yarn-daemon.sh start nodemanager

    3、配置service文件

    文件1:rexel-hdfs-journalnode.service

    [Unit]
    Description=HDFS Journalnode Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/hadoop-root-journalnode.pid
    ExecStart=/home/radmin/rexel-dev/rexel-hdfs-journalnode.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    文件2:rexel-hdfs-namenode.service

    [Unit]
    Description=HDFS Namenode Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/hadoop-root-namenode.pid
    ExecStart=/home/radmin/rexel-dev/rexel-hdfs-namenode.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    文件3:rexel-hdfs-datanode.service

    [Unit]
    Description=HDFS Datanode Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/hadoop-root-datanode.pid
    ExecStart=/home/radmin/rexel-dev/rexel-hdfs-datanode.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    文件4:rexel-hdfs-zkfc.service

    [Unit]
    Description=HDFS ZKFC Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/hadoop-root-zkfc.pid
    ExecStart=/home/radmin/rexel-dev/rexel-hdfs-zkfc.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    文件5:rexel-yarn-resourcemanager.service

    [Unit]
    Description=Yarn Resourcemanager Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/yarn-root-resourcemanager.pid
    ExecStart=/home/radmin/rexel-dev/rexel-yarn-resourcemanager.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    文件6:rexel-yarn-nodemanager.service

    [Unit]
    Description=Yarn Nodemanager Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/hadoop/yarn-root-nodemanager.pid
    ExecStart=/home/radmin/rexel-dev/rexel-yarn-nodemanager.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    4、配置软连接

    命令:

    ln -s /home/radmin/rexel-dev/rexel-hdfs-journalnode.service /usr/lib/systemd/system/rexel-hdfs-journalnode.service
    ln -s /home/radmin/rexel-dev/rexel-hdfs-namenode.service /usr/lib/systemd/system/rexel-hdfs-namenode.service
    ln -s /home/radmin/rexel-dev/rexel-hdfs-zkfc.service /usr/lib/systemd/system/rexel-hdfs-zkfc.service
    ln -s /home/radmin/rexel-dev/rexel-hdfs-datanode.service /usr/lib/systemd/system/rexel-hdfs-datanode.service
    ln -s /home/radmin/rexel-dev/rexel-yarn-resourcemanager.service /usr/lib/systemd/system/rexel-yarn-resourcemanager.service
    ln -s /home/radmin/rexel-dev/rexel-yarn-nodemanager.service /usr/lib/systemd/system/rexel-yarn-nodemanager.service

    5、加载生效

    命令:

    systemctl daemon-reload

    6、启动服务

    命令:

    systemctl start rexel-hdfs-journalnode.service
    systemctl start rexel-hdfs-namenode.service
    systemctl start rexel-hdfs-zkfc.service
    systemctl start rexel-hdfs-datanode.service
    systemctl start rexel-yarn-resourcemanager.service
    systemctl start rexel-yarn-nodemanager.service

    【小插曲】

    和zookeeper一样,提示了找不到pid文件的错误。

    解决办法:在系统环境变量中配置HADOOP_PID_DIR,YARN_PID_DIR然后修改一下service文件中pid文件的路径,重新启动服务即可。

    ==Redis==

    1、文件规划

    rexel-redis.service
    rexel-redis.service.run

    2、配置run文件

    文件:rexel-redis.service.run

    #!/bin/bash
    source /etc/profile
    /home/radmin/redis-5.0.8/src/redis-server /home/radmin/redis-5.0.8/redis.conf

    3、配置service文件

    文件:rexel-redis.service

    [Unit]
    Description=Redis Service
    After=network.target
    
    [Service]
    Type=forking
    PIDFile=/home/radmin/data/pids/redis_8793.pid
    ExecStart=/home/radmin/rexel-dev/rexel-redis.service.run
    Restart=always
    
    [Install]
    WantedBy=multi-user.target

    【小贴士】

    redis设置了默认的pid文件路径:/var/run/redis_6379.pid,为了方便管理,我重新定义了pidfile的配置。

    相应的在rexel-redis.service文件中的

    4、配置软连接

    命令:

    ln -s /home/radmin/rexel-dev/rexel-redis.service /usr/lib/systemd/system/rexel-redis.service

    5、加载生效

    命令:

    systemctl daemon-reload

    6、启动服务

    命令:

    systemctl start rexel-redis.service

    --END--

  • 相关阅读:
    【分布计算环境学习笔记】4 Enterprise Java Bean
    超详细的秒杀架构设计,运维,了解一下【转】
    Redis的监控指标【转】
    Windows netstat 查看端口、进程占用 查看进程路径
    wireshark抓包新手使用教程【转】
    关于设置sftp 指定端口【转】
    简单聊一聊Ansible自动化运维【转】
    tomcat启动报错SEVERE: Exception loading sessions from persistent storage【转】
    彻底搞懂 Kubernetes 的底层网络,看这几张图就够了【转】
    Java设计模式之(五)——代理模式
  • 原文地址:https://www.cnblogs.com/quchunhui/p/13265747.html
Copyright © 2020-2023  润新知