Hbase单节点Standalone & Pseudo Distribute

Hbase单节点Standalone & Pseudo Distribute

官方网站:

http://hbase.apache.org/

http://hbase.apache.org/book.html

环境:

CentOS 7.1

hadoop-2.7.2

hbase-1.2.1

jdk-1.8

说明:仅个人测试学习记录，大神们可以一笑而过

一.jdk

jdk版本兼容性测试，请看http://hbase.apache.org/book.html#java

二.安装hbase

1.创建用户

useradd -r -m hbadmin

echo hbadmin:hb.123|chpasswd

说明:服务启停都由hbadmin用户来操作

2.调整资源限制

cat >/etc/security/limits.d/99-hbase-nproc.conf <<HERE

hbadmin soft fsize unlimited 

hbadmin soft cpu unlimited 

hbadmin soft as unlimited 

hbadmin soft nofile 64000 

hbadmin soft nproc 64000 

hbadmin hard fsize unlimited 

hbadmin hard cpu unlimited 

hbadmin hard as unlimited 

hbadmin hard nofile 64000 

hbadmin hard nproc 64000 

HERE

3.安装

wget http://mirrors.cnnic.cn/apache/hbase/1.2.1/hbase-1.2.1-bin.tar.gz

tar -xvf hbase-1.2.1-bin.tar.gz -C /opt

mv /opt/hbase-1.2.1/ /opt/hbase

chown -R hbadmin: /opt/hbase

三.环境变量

cat >/etc/profile.d/hbase.sh <<HERE

export JAVA_HOME=/opt/jdk

export HBASE_HOME=/opt/hbase

export HBASE_CONF_DIR=/opt/hbase/conf

export HBASE_PID_DIR=/opt/hbase/run

HERE

提示:还可以直接修改/opt/hbase/conf/hbase-env.sh

四.配置

Standalone模式

1.修改hbase主配置文件

conf/hbase-site.xml

指定的目录hbase会自动创建，无需手动另建,这里指定到/home/hbadmin

2.启动hbase

/opt/hbase/bin/start-hbase.sh

[hbadmin@node-a ~]$ /opt/hbase/bin/start-hbase.sh 

starting master, logging to /opt/hbase/bin/../logs/hbase-hbadmin-master-node-a.example.com.out

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

[hbadmin@node-a ~]$ /opt/jdk/bin/jps

5692 HMaster

6062 Jps

[hbadmin@node-a ~]$ ls /home/hbadmin/

hbase  zookeeper

3.hbase shell

[hbadmin@node-a ~]$/opt/hbase/bin/hbase shell

2016-05-30 11:26:23,230 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

HBase Shell; enter 'help' for list of supported commands.

Type "exit" to leave the HBase Shell

Version 1.2.1, r8d8a7107dc4ccbf36a92f64675dc60392f85c015, Wed Mar 30 11:19:21 CDT 2016

hbase(main):002:0> create 'test','cf'

0 row(s) in 1.5690 seconds

=> Hbase::Table - test

hbase(main):003:0> list 'test'

TABLE                                                                                                                

test                                                                                                                 

1 row(s) in 0.0290 seconds

=> ["test"]

hbase(main):004:0> put 'test','row1','cf:a','value1'

0 row(s) in 0.3210 seconds

hbase(main):005:0> put 'test','row2','cf:b','value2'

0 row(s) in 0.0090 seconds

hbase(main):006:0> put 'test','row3','cf:c','value3'

0 row(s) in 0.0040 seconds

hbase(main):007:0> scan 'test'

ROW                            COLUMN+CELL                                                                           

 row1                          column=cf:a, timestamp=1464578876525, value=value1                                    

 row2                          column=cf:b, timestamp=1464578889386, value=value2                                    

 row3                          column=cf:c, timestamp=1464578898035, value=value3                                    

3 row(s) in 0.0390 seconds

hbase(main):008:0> get 'test', 'row1'

COLUMN                         CELL                                                                                  

 cf:a                          timestamp=1464578876525, value=value1                                                 

1 row(s) in 0.0500 seconds

hbase(main):013:0> drop 'test'

ERROR: Table test is enabled. Disable it first.

Here is some help for this command:

Drop the named table. Table must first be disabled:

  hbase> drop 't1'

  hbase> drop 'ns1:t1'

hbase(main):014:0> disable 'test'

0 row(s) in 2.2380 seconds

hbase(main):015:0> drop 'test'

0 row(s) in 1.2610 seconds

hbase(main):016:0> quit

4.停止hbase

[hbadmin@node-a ~]$/opt/hbase/bin/stop-hbase.sh 

stopping hbase.......................

Pseudo Distribute模式

1.启动hdfs

请参看Hadoop单节点集群

2.配置hbase

You do not need to create the directory in HDFS. HBase will do this for you. If you create the directory, HBase will attempt to do a migration, which is not what you want.

3.hdfs写入权限

注意: hbase的拥有人要有写入hdfs的权限,否则会在启动hbase时HMaster无法启动

2016-05-31 11:04:34,165 FATAL [node-a:16000.activeMasterManager] master.HMaster: Failed to become active 

master

org.apache.hadoop.security.AccessControlException: Permission denied: user=hbadmin, access=WRITE, inode="

/hbase":hadoop:supergroup:drwxr-xr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)

相对于本实验环境即

hbadmin用户要有hadoop用户的权限，当然，如果hbase用户直接设为hadoop用户就可以省略此步骤

su - root

gpasswd -a hbadmin hadoop

http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#chmod

su - hadoop

/opt/hadoop/bin/hdfs dfs -chown -R hadoop:hadoop /

/opt/hadoop/bin/hdfs dfs -chmod -R 775 /

提示:也可以通过acl控制

[hadoop@node-a ~]$ /opt/hadoop/bin/hdfs dfs -ls -d /

16/05/31 11:24:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

drwxrwxr-x   - hadoop hadoop          0 2016-05-31 11:16 /

4.ssh密钥认证

su - hbadmin

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

说明:/opt/hbase/bin/start-hbase.sh脚本会通过ssh连接调用，所以要密钥认证

5.启动(或重启hbase)

/opt/hbase/bin/stop-hbase.sh 

/opt/hbase/bin/start-hbase.sh 

[hbadmin@node-a ~]$/opt/hadoop/bin/hadoop fs -ls /hbase

16/05/31 11:28:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 7 items

drwxr-xr-x   - hbadmin hadoop          0 2016-05-31 11:16 /hbase/.tmp

drwxr-xr-x   - hbadmin hadoop          0 2016-05-31 11:17 /hbase/MasterProcWALs

drwxr-xr-x   - hbadmin hadoop          0 2016-05-31 11:16 /hbase/WALs

drwxr-xr-x   - hbadmin hadoop          0 2016-05-31 11:16 /hbase/data

-rw-r--r--   1 hbadmin hadoop         42 2016-05-31 11:16 /hbase/hbase.id

-rw-r--r--   1 hbadmin hadoop          7 2016-05-31 11:16 /hbase/hbase.version

drwxr-xr-x   - hbadmin hadoop          0 2016-05-31 11:16 /hbase/oldWALs

4.启动多个backup HMaster(可选)

目前最大可以1 master HMaster, 9 backup HMaster

[hbadmin@node-a ~]$/opt/hbase/bin/local-master-backup.sh start 2 3 5

starting master, logging to /opt/hbase/bin/../logs/hbase-hbadmin-2-master-node-a.example.com.out

starting master, logging to /opt/hbase/bin/../logs/hbase-hbadmin-3-master-node-a.example.com.out

starting master, logging to /opt/hbase/bin/../logs/hbase-hbadmin-5-master-node-a.example.com.out

说明:测试之用，可以在同一台主机上的多个端口启动backup HMaster，脚本是通过偏移量来确定端口的

每个HMaster都会占用3个端口(默认16010, 16020, and 16030)。backup HMaster占用端口就是在master HMaster3个端口的基础之上分别加上偏移量，如下：16012/16022/16032, 16013/16023/16033, and 16015/16025/16035

hbadmin      54321  54307 11 13:50 pts/2    00:00:22 /opt/jdk/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/opt/hbase/bin/../logs -Dhbase.log.file=hbase-root-2-master-config-p.example.com.log -Dhbase.home.dir=/opt/hbase/bin/.. -Dhbase.id.str=root-2 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster -D hbase.master.port=16002 -D hbase.master.info.port=16012 -D hbase.regionserver.port=16022 -D hbase.regionserver.info.port=16032 --backup start

hbadmin      54424  54410 11 13:50 pts/2    00:00:22 /opt/jdk/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/opt/hbase/bin/../logs -Dhbase.log.file=hbase-root-3-master-config-p.example.com.log -Dhbase.home.dir=/opt/hbase/bin/.. -Dhbase.id.str=root-3 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster -D hbase.master.port=16003 -D hbase.master.info.port=16013 -D hbase.regionserver.port=16023 -D hbase.regionserver.info.port=16033 --backup start

hbadmin      54494  54480 11 13:50 pts/2    00:00:21 /opt/jdk/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/opt/hbase/bin/../logs -Dhbase.log.file=hbase-root-5-master-config-p.example.com.log -Dhbase.home.dir=/opt/hbase/bin/.. -Dhbase.id.str=root-5 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster -D hbase.master.port=16005 -D hbase.master.info.port=16015 -D hbase.regionserver.port=16025 -D hbase.regionserver.info.port=16035 --backup start

停止backup HMaster

[hbadmin@node-a ~]$/opt/hbase/bin/local-master-backup.sh stop 2 3 5

stopping master.

stopping master.

stopping master.

5.启动additional RegionServers(可选)

RegionServer占用两个端口(默认16200,16300)和local-master-backup.sh类似，local-regionservers.sh也可以通过偏移量来启动多个RegionServers，最多可以达到99个additional RegionServers

[hbadmin@node-a ~]$/opt/hbase/bin/local-regionservers.sh start 2 3 4 5

starting regionserver, logging to /opt/hbase/bin/../logs/hbase-hbadmin-2-regionserver-node-a.example.com.out

starting regionserver, logging to /opt/hbase/bin/../logs/hbase-hbadmin-3-regionserver-node-a.example.com.out

starting regionserver, logging to /opt/hbase/bin/../logs/hbase-hbadmin-4-regionserver-node-a.example.com.out

starting regionserver, logging to /opt/hbase/bin/../logs/hbase-hbadmin-5-regionserver-node-a.example.com.out

停止additional RegionServers

[hbadmin@node-a ~]$/opt/hbase/bin/local-regionservers.sh stop 2 3 4 5

stopping regionserver.........

stopping regionserver.......

stopping regionserver...

stopping regionserver.

6.web UI

每个HMaster都有个web UI,如上就有

192.168.192.10:16010

192.168.192.10:16012

192.168.192.10:16013

192.168.192.10:16015