问题描述:
前几天一个同事来说,通过phoenix创建表失败了,一直报表存在的错误,删除也报错,然后就针对这个问题找下解决方案.
问题分析:
1.通过phoenix创建表,一直卡住不动了.创建语句如下:
CREATE TABLE SYNC_BUSINESS_INFO_BYDAY_EFFECT(
ID VARCHAR,
DAY_ID VARCHAR,
PROV_ID VARCHAR,
PROV_NAME VARCHAR,
AREA_ID VARCHAR,
AREA_NAME VARCHAR,
CITY_CODE VARCHAR,
CITY_NAME VARCHAR,
STORE_ID VARCHAR,
STORE_NAME VARCHAR,
STORE_TYPE VARCHAR,
CHANNEL_ID VARCHAR,
CHANNEL_NAME VARCHAR,
CHANNEL_TYPE VARCHAR,
CHANNEL_TYPE_NAME VARCHAR,
STAFF_ID VARCHAR,
STAFF_NAME VARCHAR,
NET_TYPE VARCHAR,
NET_NAME VARCHAR,
TRADE_NUM VARCHAR,
SYNC_FILE_NAME VARCHAR,
SYNC_DATE VARCHAR CONSTRAINT pk PRIMARY KEY (ID)
);
备注:在phoenix中,一直卡住不动了,没有任何的报错.另外,这个表示以前创建过的,但是可能是创建没有成功,客户端就断掉了.
2.从两个路出发,首先看hbase中是否有这个表
hbase shell
备注:通过这个命令进入到hbase中,list命令,发现并没有这个表.
3.进入到zkCli.sh即zookeeper客户端中
ls /hbase/table ls /hbase/table-lock
备注:发现是有这个表的记录的,所以通过以下命令进行删除:
rmr /hbase/table/SYNC_BUSINESS_INFO_BYDAY_EFFECT
rmr /hbase/table-lock/SYNC_BUSINESS_INFO_BYDAY_EFFECT
4.然后重启hbase,以为为解决这个问题,但是重新创建表还是卡住了.
5.查看hbase hmaster的日志,报下面的错误:
2018-05-03 17:02:48,930 INFO [B.defaultRpcServer.handler=29,queue=2,port=16000] master.HMaster: Client=aiprd//10.191.5.227 create 'SYNC_BUSINESS_INFO_BYDAY_EFFECT', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.hbase.index.Indexer|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2018-05-03 17:02:49,297 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYNC_BUSINESS_INFO_BYDAY_EFFECT) id=516 owner=aiprd state=ROLLEDBACK exec-time=251msec exception=org.apache.hadoop.hbase.TableExistsException: SYNC_BUSINESS_INFO_BYDAY_EFFECT
备注:一直报这个错误.
6.所以,就在网上找了一些方法
--1)比如,删/hbase目录重启hbase,zookpper,没有解决问题
--2)比如,关闭hbase,然后通过hbase clean --cleanZk删除hbase在zookeeper中的数据,没有解决问题
--3)比如,重启zookeeper,hbase,删除hbase在zookeeper中的数据,没有解决问题
备注:以上的方法,无论使用哪个,hmaster后台也一直报表存在的错误.
7.最后,没有办法继续找解决方法,有提到hbck命令,所以呢,就只能做尝试(这个命令的作用是检查一致性的),执行以下的命令:
[aiprd@host-10-191-5-225 hbase-1.2.6]$ bin/hbase hbck Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/aiprd/app/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2018-05-03 21:42:56,585 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable HBaseFsck command line options: 2018-05-03 21:42:57,137 INFO [main] util.HBaseFsck: Launching hbck 2018-05-03 21:42:57,228 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x12299890 connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=host-10-191-5-225 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_121 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/mnt/aiprd/app/java/jre 2018-05-03 21:42:57,243 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/mnt/aiprd/app/hbase-1.2.6/bin/../conf:/mnt/aiprd/app/java/lib/tools.jar:/mnt/aiprd/app/hbase-1.2.6/bin/..:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/activation-1.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/aopalliance-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/api-asn1-api-1.0.0-M20.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/api-util-1.0.0-M20.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/asm-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/avro-1.7.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-beanutils-1.7.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-beanutils-core-1.8.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-cli-1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-codec-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-collections-3.2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-configuration-1.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-daemon-1.0.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-digester-1.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-el-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-io-2.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-lang-2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-logging-1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-math-2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-math3-3.1.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-net-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/disruptor-3.3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/findbugs-annotations-1.3.9-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guava-12.0.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guice-3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-annotations-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-auth-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-client-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-hdfs-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-api-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-client-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-client-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-common-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-common-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-examples-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-external-blockcache-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-it-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-it-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-prefix-tree-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-procedure-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-protocol-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-resource-bundle-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-rest-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-server-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-server-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-shell-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-thrift-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/htrace-core-3.1.0-incubating.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/httpclient-4.2.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/httpcore-4.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jamon-runtime-2.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jasper-compiler-5.5.23.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/javax.inject-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/java-xmlbuilder-0.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jcodings-1.0.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-client-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-core-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-json-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-server-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jets3t-0.9.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jettison-1.3.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-sslengine-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/joni-2.1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jruby-complete-1.6.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsch-0.1.42.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsp-2.1-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsp-api-2.1-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/junit-4.12.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/libthrift-0.9.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/log4j-1.2.17.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/metrics-core-2.2.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/netty-all-4.0.23.Final.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/paranamer-2.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/phoenix-4.10.0-HBase-1.2-server.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/servlet-api-2.5-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/servlet-api-2.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/slf4j-api-1.7.7.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/slf4j-log4j12-1.7.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/spymemcached-2.11.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/xmlenc-0.52.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/xz-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/etc/hadoop:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/gson-2.2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/htrace-core-3.0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/paranamer-2.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/avro-1.7.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-el-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-framework-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-json-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-client-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsp-api-2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jettison-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/activation-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/xmlenc-0.52.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-digester-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsch-0.1.42.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-net-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/junit-4.11.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-nfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/javax.inject-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guice-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jettison-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/activation-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/contrib/capacity-scheduler/*.jar 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/mnt/aiprd/app/hadoop-2.6.0/lib/native 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=aiprd 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/mnt/aiprd 2018-05-03 21:42:57,245 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/mnt/aiprd/app/hbase-1.2.6 2018-05-03 21:42:57,246 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x122998900x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:42:57,271 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.225/10.191.5.225:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:42:57,280 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.225/10.191.5.225:2181, initiating session 2018-05-03 21:42:57,292 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.225/10.191.5.225:2181, sessionid = 0x1632604ecbc0027, negotiated timeout = 40000 Version: 1.2.6 Number of live region servers: 3 Number of dead region servers: 0 Master: host-10-191-5-225,16000,1525354611207 Number of backup masters: 0 Average load: 26.0 Number of requests: 0 Number of regions: 78 Number of regions in transition: 7 2018-05-03 21:42:58,006 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0 2018-05-03 21:42:58,229 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [IDX_TARGETCUSTDATA_SERIALNUMBER, SN_ORDERS, JDZMD_STOCK_DETAILS, SYN_SALE_PLUS_GOODS_DETAIL_SKU, SYSTEM.MUTEX, JDZMD_SALSER, SYN_SALE_PLUS_GOODS_DETAIL, JDZMD_AREAS, SYSTEM.SEQUENCE, OC_COMM_STOCK, PM_STORE_CHNL_RELATION_REPAIR, JDZMD_STOCK, SYSTEM.FUNCTION, SYN_SALE_PLUS_TRADE, SYNC_DEVELOP_INFO_BYDAY_EFFECT, TOTAL_COST_DETAIL, SYNC_BUSINESS_INFO_BYDAY_EFFECT01, DEPART_INCOME_COST_DETAIL, IDX_TARGETCUST_DATA_PROMO_ID, SYSTEM.STATS, PM_PRODUCT, SYNC01_BUSINESS_INFO_BYDAY_EFFECT, SYNC_TICKET_ACCOUNT_CHECK_INFO, SYN_SALE_PLUS_GOODS, SYNC_SALE_ORDER_INFO, SYNC_BUSINESS_INFO_BYYEAR_HUI_STORE, SYNC_BUSINESS_INFO_BYDAY_IDX_1, SYNC_BUSINESS_INFO_BYDAY_EFFECT02, ZMM_POPULATION, OC_COMM_BASE_INFO, EFFECT_INDEX, SYNC_DEVELOP_INFO_BYDAY, TOTAL_CURR_INCOME_DETAIL, JDZMD_ORDERS, SN_STORE, TOTAL_INCOME_DETAIL, SYN_SALE_PLUS_TRADE_DETAIL, JDZMD_BUSINESSES_AND_STORES, SYNC_COMM_STOCK_IMEI, TOTAL_LAST_INCOME_DETAIL, SYN_SALE_PLUS_STORE, RC_TOP_SALE_CATEGORY_20180103TMP, SYNC_BUSINESS_INFO_BYDAY_BAK1, SYN_SPARK_DEAL_LOG, TARGETCUST_DATA, PM_BRAND, SYNC_DEVELOP_INFO_BYDAY_BAK, RC_TOP_SALE_CATEGORY, SYNC_BUSINESS_INFO_BYDAY, SYNC_BUSINESS_INFO_BYDAY_HUI_STORE, SYNC_COMM_STOCK_BARCODE, SN_ORDERS_ITEM, SYNC_BUSINESS_INFO_BYDAY_EFFECT, SYSTEM.CATALOG, SN_STOCK, OC_ORDER_COMMODITY, hbase:namespace, ZMM2_POPULATION, PM_STORE, SYN_SALE_PLUS_EMPLOYEE, PROD_SALE, OC_ORDER_SALES, SN_EMPLOYEE_STORE, PM_STORE_CHNL_RELATION, PM_COMMODITY, PROD_SALE_TABLE] 2018-05-03 21:42:58,232 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6a66a204 connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:42:58,234 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x6a66a2040x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:42:58,239 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.227/10.191.5.227:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:42:58,240 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.227/10.191.5.227:2181, initiating session 2018-05-03 21:42:58,251 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.227/10.191.5.227:2181, sessionid = 0x3632604f9540019, negotiated timeout = 40000 2018-05-03 21:42:58,285 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-05-03 21:42:58,286 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x3632604f9540019 2018-05-03 21:42:58,291 INFO [main] zookeeper.ZooKeeper: Session: 0x3632604f9540019 closed 2018-05-03 21:42:58,291 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Number of Tables: 59 2018-05-03 21:42:58,301 INFO [main] util.HBaseFsck: Loading region directories from HDFS ... 2018-05-03 21:42:59,690 INFO [main] util.HBaseFsck: Loading region information from HDFS 2018-05-03 21:42:59,695 WARN [hbasefsck-pool1-t23] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 55c734e322ed22f355d5d27b3bd5c060, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:42:59,695 WARN [hbasefsck-pool1-t8] util.HBaseFsck: No HDFS region dir found: { meta => SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 691f074a362b80b6383d8894b659da88, NAME => 'SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:42:59,697 WARN [hbasefsck-pool1-t32] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 7f505d85980d982e5ffb2c368cd948d1, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:42:59,697 WARN [hbasefsck-pool1-t14] util.HBaseFsck: No HDFS region dir found: { meta => ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 851660046fc034c48aa58ea5a41808b9, NAME => 'ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9.', STARTKEY => '', ENDKEY => ''} .2018-05-03 21:42:59,904 WARN [hbasefsck-pool1-t34] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => cc93585a767dadcbfedff5e152c1e669, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:42:59,905 WARN [hbasefsck-pool1-t21] util.HBaseFsck: No HDFS region dir found: { meta => ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => de0beed9391b6d6ab68dcfd92697b662, NAME => 'ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:42:59,909 WARN [hbasefsck-pool1-t25] util.HBaseFsck: No HDFS region dir found: { meta => PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => eeb0756bc3c35f22d510502624984ea8, NAME => 'PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:43:00,393 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4e4efc1b connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:43:00,393 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x4e4efc1b0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:43:00,395 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.226/10.191.5.226:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:43:00,396 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.226/10.191.5.226:2181, initiating session 2018-05-03 21:43:00,400 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.226/10.191.5.226:2181, sessionid = 0x2632604e6f7001e, negotiated timeout = 40000 2018-05-03 21:43:00,403 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:43:00,403 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:43:00,405 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.225/10.191.5.225:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:43:00,406 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.225/10.191.5.225:2181, initiating session 2018-05-03 21:43:00,410 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.225/10.191.5.225:2181, sessionid = 0x1632604ecbc0028, negotiated timeout = 40000 2018-05-03 21:43:00,434 INFO [main] zookeeper.ZooKeeper: Session: 0x1632604ecbc0028 closed 2018-05-03 21:43:00,434 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:43:00,434 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2632604e6f7001e 2018-05-03 21:43:00,438 INFO [main] zookeeper.ZooKeeper: Session: 0x2632604e6f7001e closed 2018-05-03 21:43:00,439 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:43:00,439 INFO [main] util.HBaseFsck: Checking and fixing region consistency ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. 2018-05-03 21:43:00,619 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially. 2018-05-03 21:43:00,634 WARN [main] util.HBaseFsck: Naming new problem group: 20180124x0020180125171008535733092133537437 ERROR: (region SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1522508011325.4a14624045253764c1056f646bbe6f78.) Multiple regions have the same startkey: 20180124x0020180125171008535733092133537437 ERROR: (region SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1520317313775.7b26d838855e20824e961161d0faed68.) Multiple regions have the same startkey: 20180124x0020180125171008535733092133537437 ERROR: (regions SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1520317313775.7b26d838855e20824e961161d0faed68. and SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180226x0020180227171513028226357214718886,1522508011325.c3cb4df74c322c55a482d8f9478a0324.) There is an overlap in the region chain. 2018-05-03 21:43:00,635 WARN [main] util.HBaseFsck: reached end of problem group: null ERROR: Found inconsistency in table SYNC_BUSINESS_INFO_BYDAY_IDX_1 2018-05-03 21:43:00,636 INFO [main] util.HBaseFsck: Computing mapping of all store files .. 2018-05-03 21:43:00,902 INFO [main] util.HBaseFsck: Validating mapping using HDFS state 2018-05-03 21:43:00,918 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:43:00,918 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:43:00,919 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.225/10.191.5.225:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:43:00,920 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.225/10.191.5.225:2181, initiating session 2018-05-03 21:43:00,923 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.225/10.191.5.225:2181, sessionid = 0x1632604ecbc0029, negotiated timeout = 40000 2018-05-03 21:43:01,001 INFO [main] zookeeper.ZooKeeper: Session: 0x1632604ecbc0029 closed 2018-05-03 21:43:01,002 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:43:01,002 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:43:01,002 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:43:01,005 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.227/10.191.5.227:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:43:01,006 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.227/10.191.5.227:2181, initiating session 2018-05-03 21:43:01,010 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.227/10.191.5.227:2181, sessionid = 0x3632604f954001a, negotiated timeout = 40000 2018-05-03 21:43:01,062 INFO [main] zookeeper.ZooKeeper: Session: 0x3632604f954001a closed Summary: 2018-05-03 21:43:01,062 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Table IDX_TARGETCUSTDATA_SERIALNUMBER is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SALE_PLUS_GOODS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_TRADE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table PM_PRODUCT is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_STORE_CHNL_RELATION_REPAIR is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table JDZMD_BUSINESSES_AND_STORES is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_ORDER_COMMODITY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_TICKET_ACCOUNT_CHECK_INFO is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table TARGETCUST_DATA is okay. Number of regions: 3 Deployed on: host-10-191-5-226,16020,1525354610613 Table TOTAL_LAST_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_STORE_CHNL_RELATION is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SALE_PLUS_GOODS_DETAIL_SKU is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table EFFECT_INDEX is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table TOTAL_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_STOCK_DETAILS is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table hbase:meta is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table DEPART_INCOME_COST_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SN_ORDERS_ITEM is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SN_ORDERS is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYSTEM.CATALOG is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table PROD_SALE_TABLE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_BUSINESS_INFO_BYYEAR_HUI_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table OC_COMM_BASE_INFO is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_BUSINESS_INFO_BYDAY_BAK1 is okay. Number of regions: 2 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-227,16020,1525354610418 Table JDZMD_AREAS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table JDZMD_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_ORDERS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_BRAND is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table PM_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_SALSER is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_EMPLOYEE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_BUSINESS_INFO_BYDAY is okay. Number of regions: 5 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYNC_DEVELOP_INFO_BYDAY_BAK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYNC_DEVELOP_INFO_BYDAY is okay. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYNC_COMM_STOCK_IMEI is okay. Number of regions: 2 Deployed on: host-10-191-5-226,16020,1525354610613 Table PM_COMMODITY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SPARK_DEAL_LOG is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_ORDER_SALES is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table hbase:namespace is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table TOTAL_COST_DETAIL is okay. Number of regions: 2 Deployed on: host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_GOODS_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_SALE_ORDER_INFO is okay. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 Table SYSTEM.SEQUENCE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYSTEM.FUNCTION is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SN_EMPLOYEE_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_TRADE_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_DEVELOP_INFO_BYDAY_EFFECT is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYSTEM.MUTEX is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYSTEM.STATS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table RC_TOP_SALE_CATEGORY_20180103TMP is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_COMM_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table IDX_TARGETCUST_DATA_PROMO_ID is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_COMM_STOCK_BARCODE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SN_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SN_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_BUSINESS_INFO_BYDAY_HUI_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYNC_BUSINESS_INFO_BYDAY_IDX_1 is inconsistent. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 Table RC_TOP_SALE_CATEGORY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table TOTAL_CURR_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 10 inconsistencies detected. Status: INCONSISTENT 2018-05-03 21:43:01,075 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-05-03 21:43:01,075 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1632604ecbc0027 2018-05-03 21:43:01,078 INFO [main] zookeeper.ZooKeeper: Session: 0x1632604ecbc0027 closed 2018-05-03 21:43:01,078 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
备注:通过以上的检查信息可以知道,就是说,这些表在zookeeper的META即元数据中存在,但是在hdfs中是不存在的,所以是不一致的,才会导致建表的时候一直报表存在的错误.
8.通过hbck修复元数据的选项-fixMeta修复该问题
[aiprd@host-10-191-5-225 hbase-1.2.6]$ bin/hbase hbck -fixMeta Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/aiprd/app/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2018-05-03 21:48:44,196 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable HBaseFsck command line options: -fixMeta 2018-05-03 21:48:44,876 INFO [main] util.HBaseFsck: Launching hbck 2018-05-03 21:48:44,964 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x68dc098b connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=host-10-191-5-225 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_121 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/mnt/aiprd/app/java/jre 2018-05-03 21:48:44,970 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/mnt/aiprd/app/hbase-1.2.6/bin/../conf:/mnt/aiprd/app/java/lib/tools.jar:/mnt/aiprd/app/hbase-1.2.6/bin/..:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/activation-1.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/aopalliance-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/api-asn1-api-1.0.0-M20.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/api-util-1.0.0-M20.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/asm-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/avro-1.7.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-beanutils-1.7.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-beanutils-core-1.8.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-cli-1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-codec-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-collections-3.2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-configuration-1.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-daemon-1.0.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-digester-1.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-el-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-io-2.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-lang-2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-logging-1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-math-2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-math3-3.1.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/commons-net-3.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/disruptor-3.3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/findbugs-annotations-1.3.9-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guava-12.0.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guice-3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-annotations-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-auth-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-client-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-hdfs-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-api-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-client-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-annotations-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-client-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-common-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-common-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-examples-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-external-blockcache-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-it-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-it-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-prefix-tree-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-procedure-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-protocol-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-resource-bundle-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-rest-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-server-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-server-1.2.6-tests.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-shell-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/hbase-thrift-1.2.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/htrace-core-3.1.0-incubating.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/httpclient-4.2.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/httpcore-4.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jamon-runtime-2.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jasper-compiler-5.5.23.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/javax.inject-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/java-xmlbuilder-0.4.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jcodings-1.0.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-client-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-core-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-json-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jersey-server-1.9.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jets3t-0.9.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jettison-1.3.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-sslengine-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/joni-2.1.2.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jruby-complete-1.6.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsch-0.1.42.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsp-2.1-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/jsp-api-2.1-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/junit-4.12.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/libthrift-0.9.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/log4j-1.2.17.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/metrics-core-2.2.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/netty-all-4.0.23.Final.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/paranamer-2.3.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/phoenix-4.10.0-HBase-1.2-server.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/servlet-api-2.5-6.1.14.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/servlet-api-2.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/slf4j-api-1.7.7.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/slf4j-log4j12-1.7.5.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/spymemcached-2.11.6.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/xmlenc-0.52.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/xz-1.0.jar:/mnt/aiprd/app/hbase-1.2.6/bin/../lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/etc/hadoop:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/gson-2.2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/htrace-core-3.0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/paranamer-2.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/avro-1.7.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-el-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-framework-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-json-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/curator-client-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsp-api-2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jettison-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/activation-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/xmlenc-0.52.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-digester-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jsch-0.1.42.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-net-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/junit-4.11.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/common/hadoop-nfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/javax.inject-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guice-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jettison-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/activation-1.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/mnt/aiprd/app/hadoop-2.6.0/contrib/capacity-scheduler/*.jar 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/mnt/aiprd/app/hadoop-2.6.0/lib/native 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=aiprd 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/mnt/aiprd 2018-05-03 21:48:44,973 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/mnt/aiprd/app/hbase-1.2.6 2018-05-03 21:48:44,974 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x68dc098b0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:44,994 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.227/10.191.5.227:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:44,996 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.227/10.191.5.227:2181, initiating session 2018-05-03 21:48:45,012 INFO [main-SendThread(10.191.5.227:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.227/10.191.5.227:2181, sessionid = 0x3632604f954001b, negotiated timeout = 40000 Version: 1.2.6 Number of live region servers: 3 Number of dead region servers: 0 Master: host-10-191-5-225,16000,1525354611207 Number of backup masters: 0 Average load: 26.0 Number of requests: 0 Number of regions: 78 Number of regions in transition: 7 2018-05-03 21:48:45,783 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0 2018-05-03 21:48:45,963 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [IDX_TARGETCUSTDATA_SERIALNUMBER, SN_ORDERS, JDZMD_STOCK_DETAILS, SYN_SALE_PLUS_GOODS_DETAIL_SKU, SYSTEM.MUTEX, JDZMD_SALSER, SYN_SALE_PLUS_GOODS_DETAIL, JDZMD_AREAS, SYSTEM.SEQUENCE, OC_COMM_STOCK, PM_STORE_CHNL_RELATION_REPAIR, JDZMD_STOCK, SYSTEM.FUNCTION, SYN_SALE_PLUS_TRADE, SYNC_DEVELOP_INFO_BYDAY_EFFECT, TOTAL_COST_DETAIL, SYNC_BUSINESS_INFO_BYDAY_EFFECT01, DEPART_INCOME_COST_DETAIL, IDX_TARGETCUST_DATA_PROMO_ID, SYSTEM.STATS, PM_PRODUCT, SYNC01_BUSINESS_INFO_BYDAY_EFFECT, SYNC_TICKET_ACCOUNT_CHECK_INFO, SYN_SALE_PLUS_GOODS, SYNC_SALE_ORDER_INFO, SYNC_BUSINESS_INFO_BYYEAR_HUI_STORE, SYNC_BUSINESS_INFO_BYDAY_IDX_1, SYNC_BUSINESS_INFO_BYDAY_EFFECT02, ZMM_POPULATION, OC_COMM_BASE_INFO, EFFECT_INDEX, SYNC_DEVELOP_INFO_BYDAY, TOTAL_CURR_INCOME_DETAIL, JDZMD_ORDERS, SN_STORE, TOTAL_INCOME_DETAIL, SYN_SALE_PLUS_TRADE_DETAIL, JDZMD_BUSINESSES_AND_STORES, SYNC_COMM_STOCK_IMEI, TOTAL_LAST_INCOME_DETAIL, SYN_SALE_PLUS_STORE, RC_TOP_SALE_CATEGORY_20180103TMP, SYNC_BUSINESS_INFO_BYDAY_BAK1, SYN_SPARK_DEAL_LOG, TARGETCUST_DATA, PM_BRAND, SYNC_DEVELOP_INFO_BYDAY_BAK, RC_TOP_SALE_CATEGORY, SYNC_BUSINESS_INFO_BYDAY, SYNC_BUSINESS_INFO_BYDAY_HUI_STORE, SYNC_COMM_STOCK_BARCODE, SN_ORDERS_ITEM, SYNC_BUSINESS_INFO_BYDAY_EFFECT, SYSTEM.CATALOG, SN_STOCK, OC_ORDER_COMMODITY, hbase:namespace, ZMM2_POPULATION, PM_STORE, SYN_SALE_PLUS_EMPLOYEE, PROD_SALE, OC_ORDER_SALES, SN_EMPLOYEE_STORE, PM_STORE_CHNL_RELATION, PM_COMMODITY, PROD_SALE_TABLE] 2018-05-03 21:48:45,966 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6865c751 connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:45,966 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x6865c7510x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:45,971 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.225/10.191.5.225:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:45,971 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.225/10.191.5.225:2181, initiating session 2018-05-03 21:48:45,979 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.225/10.191.5.225:2181, sessionid = 0x1632604ecbc002a, negotiated timeout = 40000 2018-05-03 21:48:46,016 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-05-03 21:48:46,016 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1632604ecbc002a 2018-05-03 21:48:46,020 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:48:46,021 INFO [main] zookeeper.ZooKeeper: Session: 0x1632604ecbc002a closed Number of Tables: 59 2018-05-03 21:48:46,032 INFO [main] util.HBaseFsck: Loading region directories from HDFS ... 2018-05-03 21:48:46,508 INFO [main] util.HBaseFsck: Loading region information from HDFS 2018-05-03 21:48:46,513 WARN [hbasefsck-pool1-t49] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 55c734e322ed22f355d5d27b3bd5c060, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,514 WARN [hbasefsck-pool1-t20] util.HBaseFsck: No HDFS region dir found: { meta => SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 691f074a362b80b6383d8894b659da88, NAME => 'SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,515 WARN [hbasefsck-pool1-t25] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 7f505d85980d982e5ffb2c368cd948d1, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,515 WARN [hbasefsck-pool1-t28] util.HBaseFsck: No HDFS region dir found: { meta => ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => 851660046fc034c48aa58ea5a41808b9, NAME => 'ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9.', STARTKEY => '', ENDKEY => ''} .2018-05-03 21:48:46,637 WARN [hbasefsck-pool1-t7] util.HBaseFsck: No HDFS region dir found: { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => cc93585a767dadcbfedff5e152c1e669, NAME => 'SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,641 WARN [hbasefsck-pool1-t17] util.HBaseFsck: No HDFS region dir found: { meta => ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => de0beed9391b6d6ab68dcfd92697b662, NAME => 'ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,642 WARN [hbasefsck-pool1-t20] util.HBaseFsck: No HDFS region dir found: { meta => PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8., hdfs => null, deployed => , replicaId => 0 } meta={ENCODED => eeb0756bc3c35f22d510502624984ea8, NAME => 'PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8.', STARTKEY => '', ENDKEY => ''} 2018-05-03 21:48:46,968 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2755d705 connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:46,968 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hconnection-0x2755d7050x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:46,970 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.225/10.191.5.225:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:46,971 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.225/10.191.5.225:2181, initiating session 2018-05-03 21:48:46,975 INFO [main-SendThread(10.191.5.225:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.225/10.191.5.225:2181, sessionid = 0x1632604ecbc002b, negotiated timeout = 40000 2018-05-03 21:48:46,979 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:46,979 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:46,980 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.226/10.191.5.226:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:46,981 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.226/10.191.5.226:2181, initiating session 2018-05-03 21:48:46,988 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.226/10.191.5.226:2181, sessionid = 0x2632604e6f7001f, negotiated timeout = 40000 2018-05-03 21:48:47,035 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:48:47,036 INFO [main] zookeeper.ZooKeeper: Session: 0x2632604e6f7001f closed 2018-05-03 21:48:47,036 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1632604ecbc002b 2018-05-03 21:48:47,039 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:48:47,039 INFO [main] zookeeper.ZooKeeper: Session: 0x1632604ecbc002b closed 2018-05-03 21:48:47,039 INFO [main] util.HBaseFsck: Checking and fixing region consistency ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. ERROR: Region { meta => PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server. 2018-05-03 21:48:47,141 INFO [hbasefsck-pool1-t26] util.HBaseFsck: Deleted SYNC_BUSINESS_INFO_BYDAY_EFFECT01,,1524566911288.55c734e322ed22f355d5d27b3bd5c060. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t15] util.HBaseFsck: Deleted SYNC01_BUSINESS_INFO_BYDAY_EFFECT,,1524576739370.691f074a362b80b6383d8894b659da88. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t19] util.HBaseFsck: Deleted ZMM_POPULATION,,1524562845934.851660046fc034c48aa58ea5a41808b9. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t37] util.HBaseFsck: Deleted PROD_SALE,,1524567998279.eeb0756bc3c35f22d510502624984ea8. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t35] util.HBaseFsck: Deleted SYNC_BUSINESS_INFO_BYDAY_EFFECT02,,1524579217478.7f505d85980d982e5ffb2c368cd948d1. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t10] util.HBaseFsck: Deleted SYNC_BUSINESS_INFO_BYDAY_EFFECT,,1524575202010.cc93585a767dadcbfedff5e152c1e669. from META 2018-05-03 21:48:47,142 INFO [hbasefsck-pool1-t4] util.HBaseFsck: Deleted ZMM2_POPULATION,,1524576515935.de0beed9391b6d6ab68dcfd92697b662. from META 2018-05-03 21:48:47,323 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially. 2018-05-03 21:48:47,341 WARN [main] util.HBaseFsck: Naming new problem group: 20180124x0020180125171008535733092133537437 ERROR: (region SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1522508011325.4a14624045253764c1056f646bbe6f78.) Multiple regions have the same startkey: 20180124x0020180125171008535733092133537437 ERROR: (region SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1520317313775.7b26d838855e20824e961161d0faed68.) Multiple regions have the same startkey: 20180124x0020180125171008535733092133537437 ERROR: (regions SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180124x0020180125171008535733092133537437,1520317313775.7b26d838855e20824e961161d0faed68. and SYNC_BUSINESS_INFO_BYDAY_IDX_1,20180226x0020180227171513028226357214718886,1522508011325.c3cb4df74c322c55a482d8f9478a0324.) There is an overlap in the region chain. 2018-05-03 21:48:47,341 WARN [main] util.HBaseFsck: reached end of problem group: null ERROR: Found inconsistency in table SYNC_BUSINESS_INFO_BYDAY_IDX_1 2018-05-03 21:48:47,350 INFO [main] util.HBaseFsck: Computing mapping of all store files .. 2018-05-03 21:48:47,558 INFO [main] util.HBaseFsck: Validating mapping using HDFS state 2018-05-03 21:48:47,573 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:47,574 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:47,574 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.226/10.191.5.226:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:47,575 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.226/10.191.5.226:2181, initiating session 2018-05-03 21:48:47,582 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.226/10.191.5.226:2181, sessionid = 0x2632604e6f70020, negotiated timeout = 40000 2018-05-03 21:48:47,593 INFO [main] zookeeper.ZooKeeper: Session: 0x2632604e6f70020 closed 2018-05-03 21:48:47,593 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:48:47,594 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 2018-05-03 21:48:47,594 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=10.191.5.225:2181,10.191.5.226:2181,10.191.5.227:2181, baseZNode=/hbase 2018-05-03 21:48:47,595 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Opening socket connection to server 10.191.5.226/10.191.5.226:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-03 21:48:47,596 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Socket connection established to 10.191.5.226/10.191.5.226:2181, initiating session 2018-05-03 21:48:47,600 INFO [main-SendThread(10.191.5.226:2181)] zookeeper.ClientCnxn: Session establishment complete on server 10.191.5.226/10.191.5.226:2181, sessionid = 0x2632604e6f70021, negotiated timeout = 40000 2018-05-03 21:48:47,643 INFO [main] zookeeper.ZooKeeper: Session: 0x2632604e6f70021 closed 2018-05-03 21:48:47,643 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2018-05-03 21:48:47,757 INFO [main] util.HBaseFsck: Finishing hbck Summary: Table IDX_TARGETCUSTDATA_SERIALNUMBER is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SALE_PLUS_GOODS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_TRADE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table PM_PRODUCT is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_STORE_CHNL_RELATION_REPAIR is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table JDZMD_BUSINESSES_AND_STORES is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_ORDER_COMMODITY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_TICKET_ACCOUNT_CHECK_INFO is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table TARGETCUST_DATA is okay. Number of regions: 3 Deployed on: host-10-191-5-226,16020,1525354610613 Table TOTAL_LAST_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_STORE_CHNL_RELATION is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SALE_PLUS_GOODS_DETAIL_SKU is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table EFFECT_INDEX is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table TOTAL_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_STOCK_DETAILS is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table hbase:meta is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table DEPART_INCOME_COST_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SN_ORDERS_ITEM is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SN_ORDERS is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYSTEM.CATALOG is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table PROD_SALE_TABLE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_BUSINESS_INFO_BYYEAR_HUI_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table OC_COMM_BASE_INFO is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_BUSINESS_INFO_BYDAY_BAK1 is okay. Number of regions: 2 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-227,16020,1525354610418 Table JDZMD_AREAS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table JDZMD_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_ORDERS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table PM_BRAND is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table PM_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table JDZMD_SALSER is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_EMPLOYEE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_BUSINESS_INFO_BYDAY is okay. Number of regions: 5 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYNC_DEVELOP_INFO_BYDAY_BAK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYNC_DEVELOP_INFO_BYDAY is okay. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYNC_COMM_STOCK_IMEI is okay. Number of regions: 2 Deployed on: host-10-191-5-226,16020,1525354610613 Table PM_COMMODITY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYN_SPARK_DEAL_LOG is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_ORDER_SALES is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table hbase:namespace is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table TOTAL_COST_DETAIL is okay. Number of regions: 2 Deployed on: host-10-191-5-226,16020,1525354610613 host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_GOODS_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYNC_SALE_ORDER_INFO is okay. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 Table SYSTEM.SEQUENCE is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYSTEM.FUNCTION is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SN_EMPLOYEE_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYN_SALE_PLUS_TRADE_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_DEVELOP_INFO_BYDAY_EFFECT is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table SYSTEM.MUTEX is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYSTEM.STATS is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table RC_TOP_SALE_CATEGORY_20180103TMP is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table OC_COMM_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table IDX_TARGETCUST_DATA_PROMO_ID is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_COMM_STOCK_BARCODE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SN_STOCK is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SN_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-227,16020,1525354610418 Table SYNC_BUSINESS_INFO_BYDAY_HUI_STORE is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 Table SYNC_BUSINESS_INFO_BYDAY_IDX_1 is inconsistent. Number of regions: 4 Deployed on: host-10-191-5-225,16020,1525354612066 host-10-191-5-226,16020,1525354610613 Table RC_TOP_SALE_CATEGORY is okay. Number of regions: 1 Deployed on: host-10-191-5-225,16020,1525354612066 Table TOTAL_CURR_INCOME_DETAIL is okay. Number of regions: 1 Deployed on: host-10-191-5-226,16020,1525354610613 10 inconsistencies detected. Status: INCONSISTENT 2018-05-03 21:48:47,761 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2018-05-03 21:48:47,762 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x3632604f954001b 2018-05-03 21:48:47,765 INFO [main] zookeeper.ZooKeeper: Session: 0x3632604f954001b closed 2018-05-03 21:48:47,765 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
备注:通过以上的信息可以知道,已经将上面的表从META即元数据中删掉了.
9.再次在phoenix中执行表创建,没有问题,删除表也没有问题
小结:
使用hbase本身的工具可以解决很多的问题,在生产环境中要经常使用hbck检查不一致性的问题.
hbase hbck
hbase hbck -fixMeta 修复元数据的问题.
文档创建时间:2018年5月4日10:13:16