• VMWare 搭建 Hadoop 完全分布式集群(含HBASE)


    使用VMware安装CentOS6.8详细教程

    https://blog.csdn.net/weixin_44878850/article/details/89111148

    1. 虚拟机安装、克隆

    2. 网络配置

    # 修改 hostname
    hostname hadoop1
    
    hostnamectl set-hostname hadoop1
    
    # 最靠谱方式
    Vim /etc/sysconfig/network
    

    打开网卡:

    centos 安装后默认是关闭网卡的,要进 /etc/sysconfig/network-scripts/ 下修改 ifcfg-ens33,将ONBOOT 的值改成 yesONBOOT是指明在系统启动时是否激活网卡,只有在激活状态的网卡才能连接网络,进行网络通信

    修改完毕后,service network restart 重启网络,检查 ping baidu.com 是否能够通


    克隆虚拟机修改 onboot 后重启网络失败

    • 错误类型:Bringing up interface eth0:Device eth0 does not seem to be present,delaying

    • 解决方法:删除 /etc/udev/rules.d/70-persistent-net.rules,然后将网卡配置文件 /etc/sysconfig/network-scripts/ifcfg-eth0uuidhwaddr 这两行删除,重启系统即可!

    • 参考:https://jingyan.baidu.com/article/e75aca85006645142edac6df.html

    注意:有些没有 ifcfg-ens33,只有 ifcfg-eth0,修改 ifcfg-eth0 即可!

    2.1 设置静态 IP、主机映射

    设置静态 IP

    以下操作每台服务器 hadoop1/2/3 都需要设置:

    1、Vim /etc/sysconfig/network-scripts/ifcfg-ens33,有些可能会是 ifcfg-eth0

    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=static   	# 设置为 静态
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=ens33
    UUID=1c71c1dc-e4f8-4594-b77e-5e04f6906a31
    DEVICE=ens33
    ONBOOT=yes			# 必须为 yes
    IPADDR=192.168.131.137	# 静态 IP	
    GATEWAY=192.168.2.2		# 网关
    NETMAK=255.255.255.0	# 子网掩码
    DNS1=8.8.8.8
    DNS2=114.114.114.114
    

    2、每一台虚拟机配置 ip 和域名映射:

    # vim /etc/hosts
    192.168.131.137 hadoop1
    192.168.131.138 hadoop2
    192.168.131.139 hadoop3
    

    3、关闭防火墙:

    # 关闭防火墙
    Service iptables stop 
    
    # 禁止防火墙开机自启
    Chkconfig iptables off
    
    # 关闭selinux,设置 SELINUX 参数为 disabled
    Vim /etc/selinux/config
    

    4、重启网络、检查:

    # 重启网络
    service network restart
    
    # hadoop1/2/3 三台服务器之间相互 ping 看是否能够通
    ping hadoop1	
    
    # 重启
    reboot
    

    2.2 添加hadoop账户并分配sudo权限

    useradd -m hadoop
    passwd hadoop
    
    # 分配 sudo 权限,在该行root ALL=(ALL) ALL下添加hadoop ALL=(ALL) ALL保存后退出,并切换回hadoop用户
    visudo
    
    # 如下配置
    ## The COMMANDS section may have other options added to it.
    ##
    ## Allow root to run any commands anywhere
    root    ALL=(ALL)       ALL
    hadoop  ALL=(ALL)       ALL
    
    # 切换用户
    su hadoop
    

    三台服务器IP、用户名、密码对应

    # 分别有两个用户:root(110139)、hadoop(hadoop)
    192.168.131.137 hadoop1
    192.168.131.138 hadoop2
    192.168.131.139 hadoop3
    

    2.3 实现三台服务器之间免密登录

    1、安装 ssh 服务:

    yum install -y openssl openssh-server
    yum -y install openssh-clients			# 一定要安装客户端,不然使用不了 ssh
    
    # 用vim打开配置文件/etc/ssh/sshd_config
    # 将 PermitRootLogin,RSAAuthentication,PubkeyAuthentication 的注释设置打开
    
    # 常用命令
    service sshd restart # 重启SSH服务
    service sshd start # 启动服务
    service sshd stop  # 停止服务
    netstat -antp | grep sshd 	# 查看是否启动22端口
    
    chkconfig sshd on		# 设置开机启动
    chkconfig sshd off	# 设置禁止开机启动
    

    yum install 时出现:

    [root@hadoop3 .ssh]# yum install -y openssl openssh-server
    Loaded plugins: fastestmirror
    Setting up Install Process
    Loading mirror speeds from cached hostfile
    YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
     Eg. Invalid release/repo/arch combination/
    removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
    Error: Cannot find a valid baseurl for repo: base
    

    解决方法:

    sed -i "s|enabled=1|enabled=0|g" /etc/yum/pluginconf.d/fastestmirror.conf
    mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
    curl -o /etc/yum.repos.d/CentOS-Base.repo https://www.xmpan.com/Centos-6-Vault-Aliyun.repo 
        
    yum clean all
    yum makecache
    yum repolist		# 查看当前源
    

    参考:YumRepo Error: All mirror URLs are not using ftp, http[s] or file.centos6 yum失败!

    2、生成公钥私钥(一定要在 hadoop 用户下执行!!!,否则后续 start-dfs.sh 没有权限

    # 一路回车即可 路径:/home/hadoop/.ssh/id_rsa、id_rsa.pub
    ssh-keygen -t rsa
    
    # 在 hadoop1  服务器
    cd /home/hadoop/.ssh/
    cat id_rsa.pub > authorized_keys
    
    # 再将 Hadoop2、Hadoop3 的 id_rsa.pub 的内容拷贝到 Hadoop1 的 authorized_keys,最后将 authorized_keys 分别传到 Hadoop2、Hadoop3 的 /home/hadoop/.ssh/
    scp authorized_keys root@hadoop2:/home/hadoop/.ssh/
    scp authorized_keys root@hadoop3:/home/hadoop/.ssh/
        
    # 分别给三个节点 修改authorized_keys权限
    chmod 600 authorized_keys
    chmod 700 .ssh
    
    # 测试,ssh + 主机名
    ssh hadoop2
    
    # hbase 集群
    https://www.cnblogs.com/qingyunzong/p/8668880.html
    
    # hadoop 集群
    https://www.cnblogs.com/qingyunzong/p/8496127.html
    

    3. jdk 安装

    1、如果你使用root用户进行安装。 vi /etc/profile 即可 系统变量

    2、如果你使用普通用户进行安装。 vi ~/.bashrc 用户变量

    # .bashrc
    
    # Source global definitions
    if [ -f /etc/bashrc ]; then
    	. /etc/bashrc
    fi
    
    # User specific aliases and functions
    export JAVA_HOME=/home/hadoop/apps/jdk1.8.0_261
    export PATH=$JAVA_HOME/bin:$PATH
    

    检查:

    [hadoop@hadoop3 apps]$ source ~/.bashrc
    [hadoop@hadoop3 apps]$ echo $JAVA_HOME
    /home/hadoop/apps/jdk1.8.0_261
    [hadoop@hadoop3 apps]$ java -version
    java version "1.8.0_261"
    Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
    Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
    

    4. hadoop 安装

    集群规划:

    服务/主机名 IP 用户 HDFS YARN
    hadoop1 192.168.131.137 hadoop NameNode,DataNode NodeManager,ResourceManager
    hadoop2 192.168.131.138 hadoop DataNode NodeManager
    hadoop3 192.168.131.139 hadoop DataNode NodeManager
    服务 主节点 从节点
    HDFS NameNode DataNode
    YARN ResourceManager NodeManager

    注意:hadoop 用户最好给 apps、data、hadoop-2.7.5/ 等都添加权限 sudo chown hadoop:hadoop apps/

    1、解压:

    [hadoop@hadoop1 apps]$ tar -zxvf hadoop-2.7.5.tar.gz
    

    2、hadoop-env.sh

    配置文件目录:/home/hadoop/apps/hadoop-2.7.5/etc/hadoop

    cd /home/hadoop/apps/hadoop-2.7.5/etc/hadoop
    vim hadoop-env.sh
    
    # 只需修改 jdk 路径
    export JAVA_HOME=/usr/local/jdk1.8.0_73
    

    3、core-site.xml

    <configuration>
            <property>
                    <name>fs.defaultFS</name>
                    <value>hdfs://hadoop1:9000</value>
            </property>
            <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/home/hadoop/data/hadoopdata</value>
            </property>
    </configuration>
    

    4、hdfs-site.xml

    <configuration>
            <property>
                    <name>dfs.namenode.name.dir</name>
                    <value>/home/hadoop/data/hadoopdata/name</value>
                    <description>为了保证元数据的安全一般配置多个不同目录</description>
            </property>
    
            <property>
                    <name>dfs.datanode.data.dir</name>
                    <value>/home/hadoop/data/hadoopdata/data</value>
                    <description>datanode 的数据存储目录</description>
            </property>
    
            <property>
                    <name>dfs.replication</name>
                    <value>2</value>
                    <description>HDFS 的数据块的副本存储个数, 默认是3</description>
            </property>
    
            <property>
                    <name>dfs.secondary.http.address</name>
                    <value>hadoop3:50090</value>
                    <description>secondarynamenode 运行节点的信息,和 namenode 不同节点</description>
            </property>
    </configuration>
    

    5、mapred-site.xml

    <configuration>
            <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
            </property>
    </configuration>
    

    6、yarn-site.xml

    <configuration>
    
    <!-- Site specific YARN configuration properties -->
    
            <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>hadoop1</value>			// 这个必须与 resourcemanager 服务器一致,我这里是 hadoop1
            </property>
            
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
                    <description>YARN 集群为 MapReduce 程序提供的 shuffle 服务</description>
            </property>
    
    </configuration>
    

    7、slaves

    vim slaves 
    
    hadoop1
    hadoop2
    hadoop3
    

    8、将 hadoop-2.7.5/ 分发到其他集群:

    scp -r hadoop-2.7.5/ hadoop2:/home/hadoop/apps
    scp -r hadoop-2.7.5/ hadoop3:/home/hadoop/apps
    

    9、添加环境变量(每台服务器都需要添加):

    [hadoop@hadoop1 apps]$ vim ~/.bashrc 
    export HADOOP_HOME=/home/hadoop/apps/hadoop-2.7.5
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:
                
    [hadoop@hadoop1 apps]$ source ~/.bashrc 
    [hadoop@hadoop1 apps]$ hadoop version
    Hadoop 2.7.5
    Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 18065c2b6806ed4aa6a3187d77cbe21bb3dba075
    Compiled by kshvachk on 2017-12-16T01:06Z
    Compiled with protoc 2.5.0
    From source with checksum 9f118f95f47043332d51891e37f736e9
    This command was run using /home/hadoop/apps/hadoop-2.7.5/share/hadoop/common/hadoop-common-2.7.5.jar
    

    10、格式化:

    [hadoop@hadoop1 bin]$ hadoop namenode -format
    
    21/06/13 14:34:26 INFO util.GSet: Computing capacity for map NameNodeRetryCache
    21/06/13 14:34:26 INFO util.GSet: VM type       = 64-bit
    21/06/13 14:34:26 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
    21/06/13 14:34:26 INFO util.GSet: capacity      = 2^15 = 32768 entries
    21/06/13 14:34:27 INFO namenode.FSImage: Allocated new BlockPoolId: BP-57205935-192.168.131.137-1623566067094
    21/06/13 14:34:27 INFO common.Storage: Storage directory /home/hadoop/data/hadoopdata/name has been successfully formatted.
    21/06/13 14:34:27 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 using no compression
    21/06/13 14:34:27 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 of size 322 bytes saved in 0 seconds.
    21/06/13 14:34:28 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    21/06/13 14:34:28 INFO util.ExitUtil: Exiting with status 0
    21/06/13 14:34:28 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop1/192.168.131.137
    ************************************************************/
    

    格式化 Namenode 时不能创建目录

    java.io.IOException: Cannot create directory /home/hadoop/data/hadoopdata/na
        
    # 解决方法
    https://blog.csdn.net/qq_40414738/article/details/99544777
    

    start-dfs.sh 时没有权限

    [hadoop@hadoop1 sbin]$ sudo start-dfs.sh
    [sudo] password for hadoop: 
    sudo: start-dfs.sh: command not found
    [hadoop@hadoop1 sbin]$ start-dfs.sh
    Starting namenodes on [hadoop1]
    hadoop@hadoop1's password: 
    hadoop1: namenode running as process 4090. Stop it first.
    hadoop@hadoop2's password: hadoop@hadoop1's password: hadoop@hadoop3's password: 
    hadoop2: mkdir: cannot create directory `/home/hadoop/apps/hadoop-2.7.5/logs': Permission denied
    hadoop2: chown: cannot access `/home/hadoop/apps/hadoop-2.7.5/logs': No such file or directory
    hadoop2: starting datanode, logging to /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out
    hadoop2: /home/hadoop/apps/hadoop-2.7.5/sbin/hadoop-daemon.sh: line 159: /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out: No such file or directory
    hadoop2: head: cannot open `/home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out' for reading: No such file or directory
    hadoop2: /home/hadoop/apps/hadoop-2.7.5/sbin/hadoop-daemon.sh: line 177: /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out: No such file or directory
    hadoop2: /home/hadoop/apps/hadoop-2.7.5/sbin/hadoop-daemon.sh: line 178: /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out: No such file or directory
    

    解决办法:https://www.cnblogs.com/zknublx/p/8066693.html

    开启 hdfs

    [hadoop@hadoop1 sbin]$ start-dfs.sh 
    Starting namenodes on [hadoop1]
    hadoop1: starting namenode, logging to /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-namenode-hadoop1.out
    hadoop3: starting datanode, logging to /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop3.out
    hadoop2: starting datanode, logging to /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-datanode-hadoop2.out
    hadoop1: datanode running as process 4822. Stop it first.
    Starting secondary namenodes [hadoop3]
    hadoop3: starting secondarynamenode, logging to /home/hadoop/apps/hadoop-2.7.5/logs/hadoop-hadoop-secondarynamenode-hadoop3.out
    [hadoop@hadoop1 sbin]$ jps
    4822 DataNode
    5465 NameNode
    5724 Jps
    

    开启 yarn

    [hadoop@hadoop1 sbin]$ start-yarn.sh
    starting yarn daemons
    starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.7.5/logs/yarn-hadoop-resourcemanager-hadoop1.out
    hadoop3: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.7.5/logs/yarn-hadoop-nodemanager-hadoop3.out
    hadoop2: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.7.5/logs/yarn-hadoop-nodemanager-hadoop2.out
    hadoop1: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.7.5/logs/yarn-hadoop-nodemanager-hadoop1.out
    

    查看集群 jps 进程

    [hadoop@hadoop1 sbin]$ jps
    5891 Jps
    4822 DataNode
    5465 NameNode
    5770 ResourceManager
    5870 NodeManager
    
    [hadoop@hadoop2 ~]$ jps
    2643 NodeManager
    2532 DataNode
    2717 Jps
    
    [hadoop@hadoop3 ~]$ jps
    2816 DataNode
    2994 NodeManager
    3091 Jps
    

    web 管理界面

    https://www.cnblogs.com/qingyunzong/p/8496127.html
    https://www.cnblogs.com/qingyunzong/p/8668880.html
    https://blog.csdn.net/Beans___Lee/article/details/105265244
    https://blog.csdn.net/qq_43106863/article/details/101530233
    https://blog.csdn.net/ASN_forever/article/details/80701003
    https://blog.csdn.net/weixin_41552767/article/details/107221454
    https://segmentfault.com/a/1190000038229319
    VMware 搭建 hadoop 完全分布式集群
    

    5. zookeeper 集群安装

    这里规划三台服务器:hadoop1,hadoop2,hadoop3

    下载地址:http://mirrors.hust.edu.cn/apache/ZooKeeper/

    此处使用的是3.4.10版本

    1、解压:

    [hadoop@hadoop1 ~]$ cd /home/hadoop/apps
    [hadoop@hadoop1 ~]$ tar -zxvf zookeeper-3.4.10.tar.gz -C apps/
    

    2、修改配置文件:

    [hadoop@hadoop1 zookeeper-3.4.10]$ cd conf/
    [hadoop@hadoop1 conf]$ mv zoo_sample.cfg zoo.cfg
    [hadoop@hadoop1 conf]$ vim zoo.cfg 
    
    # dataDir  内存数据库快照存放地址
    dataDir=/home/hadoop/data/zkdata/
    
    # 末尾添加,配置ZK监听客户端连接的端口
    dataLogDir=/home/hadoop/log/zklog		# 事务日志
    
    server.1=hadoop1:2888:3888
    server.2=hadoop2:2888:3888
    server.3=hadoop3:2888:3888
            
    # server.serverid=host:tickpot:electionport   
    # server:固定写法
    # serverid:每个服务器的指定ID(必须处于1-255之间,必须每一台机器不能重复)
    # host:主机名
    # tickpot:心跳通信端口
    # electionport:选举端口
    

    3、将配置文件分发到集群其他机器中:

    [hadoop@hadoop1 bin]$ scp -r zookeeper-3.4.10/ hadoop2:/home/hadoop/apps
    [hadoop@hadoop1 bin]$ scp -r zookeeper-3.4.10/ hadoop3:/home/hadoop/apps
    

    注意:以下操作 4、5、6 每台服务器都需要这样操作!

    4、创建日志目录、配置服务器 ID:

    [hadoop@hadoop1 bin]$ mkdir -p /home/hadoop/log
    [hadoop@hadoop1 bin]$ sudo chown hadoop:hadoop log/			# 给 hadoop 用户添加权限
    [hadoop@hadoop1 bin]$ mkdir -p /home/hadoop/data/zkdata		
    [hadoop@hadoop1 ~]$ cd /home/hadoop/data/zkdata
    [hadoop@hadoop1 ~]$ echo 1 > myid		# myid,里面存放的内容就是服务器的 id,就是 server.1=hadoop01:2888:3888 当中的 id, 就是 1
    

    5、配置环境变量:

    vim ~/.bashrc
    
    export ZOOKEEPER_HOME=/home/hadoop/apps/zookeeper-3.4.10
    export PATH=$PATH:$ZOOKEEPER_HOME/bin
    
    source ~/.bashrc
    

    6、启动 zkserver 服务,检查:

    启动:zkServer.sh start
    停止:zkServer.sh stop
    查看状态:zkServer.sh status
    
    ps -aux | grep 'zookeeper' 
    
    [hadoop@hadoop1 bin]$ zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [hadoop@hadoop1 bin]$ zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Error contacting service. It is probably not running.
    [hadoop@hadoop1 bin]$ jps
    2769 Jps
    2723 QuorumPeerMain
    
    [hadoop@hadoop2 bin]$ zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [hadoop@hadoop2 bin]$ zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: leader
    [hadoop@hadoop2 bin]$ jps
    
    [hadoop@hadoop3 bin]$ zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [hadoop@hadoop3 bin]$ zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: follower
    [hadoop@hadoop3 bin]$ jps
    1898 QuorumPeerMain
    1951 Jps
    

    参考:https://www.cnblogs.com/qingyunzong/p/8619184.html

    6. HBase 集群搭建

    [hadoop@hadoop1 apps]$ tar -zxvf hbase-1.2.6-bin.tar.gz
    [hadoop@hadoop1 apps]$ cd hbase-1.2.6/conf/				# 配置文件目录
    
    [hadoop@hadoop1 conf]$ vim hbase-env.sh 
    
    # 修改 jdk 路径 和 zookeeper 
    export JAVA_HOME=/home/hadoop/apps/jdk1.8.0_261
    
    # Tell HBase whether it should manage it's own instance of Zookeeper or not.
    export HBASE_MANAGES_ZK=false
    

    1、修改 hbase-site.xml

    <configuration>
            <property>
                    <!-- 指定 hbase 在 HDFS 上存储的路径 -->
                    <name>hbase.rootdir</name>
                    <value>hdfs://hadoop1:9000/hbase</value>
            </property>
            <property>
                    <!-- 指定 hbase 是分布式的 -->
                    <name>hbase.cluster.distributed</name>
                    <value>true</value>
            </property>
            <property>
                    <!-- 指定 zk 的地址,多个用“,”分割 -->
                    <name>hbase.zookeeper.quorum</name>
                    <value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
            </property>
            <property>
                <name>hbase.master.maxclockskew</name>
                <value>180000</value>
                <description>Time difference of regionserver from master</description>
           </property>
    </configuration>
    

    2、修改 regionservers

    vim regionservers
    
    hadoop1
    hadoop2
    hadoop3
    

    3、修改 backup-masters(该文件是不存在的,先自行创建):

    # 备用 master
    vim backup-masters
    
    hadoop3
    

    4、要把 hadoop 的 hdfs-site.xml 和 core-site.xml 放到 hbase-1.2.6/conf 下

    5、删除 docs(删除 HBase 目录下的 docs 文件夹):

    rm -rf docs
    
    # 分发
    scp -r hbase-1.2.6/ hadoop2:/home/hadoop/apps/
    scp -r hbase-1.2.6/ hadoop3:/home/hadoop/apps/
    

    6、时间同步:

    HBase 集群对于时间的同步要求的比 HDFS 严格,所以,集群启动之前千万记住要进行 时间同步,要求相差不要超过 30s:

    方法一:

    # 配置 hbase-site.xml,三个服务器之间时间差不超过 30s
    
    <property>
        <name>hbase.master.maxclockskew</name>
        <value>180000</value>
        <description>Time difference of regionserver from master</description>
    </property>
    

    方法二:

    • 从节点服务器与主节点服务器时间同步
    • 所有节点与网络服务器时间同步
    [hadoop@hadoop2 hbase-1.2.6]$ sudo yum install -y ntp
    
    # 与 time.nist.gov 网络服务器同步
    [hadoop@hadoop2 hbase-1.2.6]$ sudo /usr/sbin/ntpdate time.nist.gov
    15 Jun 20:54:27 ntpdate[1780]: step time server 132.163.97.3 offset -28785.531070 sec
                
    # 设置定时任务
    [hadoop@hadoop2 hbase-1.2.6]$ crontab -l
    # 服务器时间同步
    0 */1 * * *  /usr/sbin/ntpdate time.nist.gov
    

    参考:http://www.hnbian.cn/posts/f2bc4737.html

    7、系统环境变量:

    # 所有节点都需要修改
    [hadoop@hadoop1 bin]$ vim ~/.bashrc
    
    export HBASE_HOME=/home/hadoop/apps/hbase-1.2.6
    export PATH=$PATH:$HBASE_HOME/bin
        
    source ~/.bashrc 
    

    8、启动 hbase 集群:

    [hadoop@hadoop1 bin]$ pwd
    /home/hadoop/apps/hbase-1.2.6/bin
    [hadoop@hadoop1 bin]$ start-hbase.sh
    starting master, logging to /home/hadoop/apps/hbase-1.2.6/logs/hbase-hadoop-master-hadoop1.out
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
    hadoop3: starting regionserver, logging to /home/hadoop/apps/hbase-1.2.6/logs/hbase-hadoop-regionserver-hadoop3.out
    hadoop2: starting regionserver, logging to /home/hadoop/apps/hbase-1.2.6/logs/hbase-hadoop-regionserver-hadoop2.out
    hadoop1: starting regionserver, logging to /home/hadoop/apps/hbase-1.2.6/logs/hbase-hadoop-regionserver-hadoop1.out
    hadoop3: starting master, logging to /home/hadoop/apps/hbase-1.2.6/logs/hbase-hadoop-master-hadoop3.out
    

    注意:在哪个集群启动,哪个就是主节点 Master,启动 hbase 之前一定先启动 zookeeper、hadoop!!!

    9、检查:

    [hadoop@hadoop1 bin]$ jps
    1668 NameNode
    2164 NodeManager
    4054 Jps
    3590 HMaster
    1495 QuorumPeerMain
    3719 HRegionServer
    1768 DataNode
    2061 ResourceManager
    
    [hadoop@hadoop2 hbase-1.2.6]$ jps
    1440 QuorumPeerMain
    2305 Jps
    2149 HRegionServer
    1622 NodeManager
    1511 DataNode
    
    [hadoop@hadoop3 bin]$ jps
    2722 Jps
    1682 NodeManager
    1436 QuorumPeerMain
    1501 DataNode
    2654 HRegionServer
    

    若有节点 HRegionServer 进程没起来就手动启动

    # 启动 master 进程 
    hbase-daemon.sh start master
    
    # 启动HRegionServer进程
    hbase-daemon.sh start regionserver 
    

    10、web-ui

    http://192.168.131.137:16010/

    11、测试:

    [hadoop@hadoop1 bin]$ hbase shell
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/home/hadoop/apps/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/home/hadoop/apps/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017
    
    hbase(main):001:0> list
    TABLE                                                                                                               
    0 row(s) in 0.6420 seconds
    
    => []
    hbase(main):002:0> helo 'list'
    NoMethodError: undefined method `helo' for #<Object:0x3ab35b9c>
    
    
    hbase(main):004:0> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
    0 row(s) in 1.6840 seconds
    
    => Hbase::Table - t1
    hbase(main):005:0> list
    TABLE                                                                                                               
    t1                                                                                                                  
    1 row(s) in 0.0290 seconds
    
    => ["t1"]
    hbase(main):006:0> desc 't1'
    Table t1 is ENABLED                                                                                                 
    t1                                                                                                                  
    COLUMN FAMILIES DESCRIPTION                                                                                         
    {NAME => 'f1', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOC
    K_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE 
    => '65536', REPLICATION_SCOPE => '0'}                                                                               
    {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOC
    K_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE 
    => '65536', REPLICATION_SCOPE => '0'}                                                                               
    {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOC
    K_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE 
    => '65536', REPLICATION_SCOPE => '0'}                                                                               
    3 row(s) in 0.3510 seconds
    

    参考文章:

  • 相关阅读:
    jBox使用方法
    第五章 引用类型
    第四章 变量、作用域、内存问题
    第二章 在HTML中使用JavaScript
    第三章 基本概念
    第一章 JavaScript简介
    ecmall 添加自定义挂件以及解决一直loading的问题
    急急如律令!火速搭建一个C#即时通信系统!(附源码分享——高度可移植!)
    委托笔记
    Oracle ORA-01461:仅可以为插入LONG列的LONG值赋值
  • 原文地址:https://www.cnblogs.com/midworld/p/15143573.html
Copyright © 2020-2023  润新知