• hbase0.92 数据修复


    现有集群环境;8台region server、1台master server

    早晨重启某一region server后,该server的region全部下线未恢复。

    重启前

    hbase(main):031:0> status 'simple'
    7 live servers
        ec2d-cntprocdn-03.mypna.com:60020 1556378718902
            requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480
        ec2d-cntprocdn-05.mypna.com:60020 1556378795513
            requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480
        ec2d-cntprocdn-04.mypna.com:60020 1556378718227
            requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480
        ec2d-cntprocdn-06.mypna.com:60020 1556378767123
            requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480
        ec2d-cntprocdn-01.mypna.com:60020 1556379153372
            requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480
        ec2d-cntprocdn-02.mypna.com:60020 1556379080369
            requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480
        ec2d-cntprocdn-07.mypna.com:60020 1556587728098
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480
    0 dead servers
    Aggregate load: 0, regions: 442

    将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线

    hbase(main):009:0> status 'simple'
    7 live servers
        ec2d-cntprocdn-03.mypna.com:60020 1556378718902
            requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480
        ec2d-cntprocdn-05.mypna.com:60020 1556378795513
            requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480
        ec2d-cntprocdn-04.mypna.com:60020 1556378718227
            requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480
        ec2d-cntprocdn-06.mypna.com:60020 1556378767123
            requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480
        ec2d-cntprocdn-01.mypna.com:60020 1556588642407
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480
        ec2d-cntprocdn-02.mypna.com:60020 1556379080369
            requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480
        ec2d-cntprocdn-07.mypna.com:60020 1556587728098
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480
    0 dead servers
    Aggregate load: 9, regions: 48

    通过 status 'detailed'查看详细region信息

    hbase(main):032:0> status 'detailed'
    version 0.92.1
    3 regionsInTransition
        .META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null
        vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null
        vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null
    master coprocessors: []

    发现元数据表'.META.' region无法正常上线

    于是常识重启集群后再次查看状态,发现所有server的region下线

    hbase(main):025:0> status 'simple'
    7 live servers
        ec2d-cntprocdn-07.mypna.com:60020 1556589413648
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480
        ec2d-cntprocdn-06.mypna.com:60020 1556589532168
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480
        ec2d-cntprocdn-05.mypna.com:60020 1556589462880
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480
        ec2d-cntprocdn-02.mypna.com:60020 1556589822614
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480
        ec2d-cntprocdn-03.mypna.com:60020 1556589365893
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=131, maxHeapMB=4480
        ec2d-cntprocdn-01.mypna.com:60020 1556589914551
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480
        ec2d-cntprocdn-04.mypna.com:60020 1556589364969
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480
    0 dead servers
    Aggregate load: 0, regions: 0

    查看表内容直接报“-ROOT-”表下线

    base(main):034:0> scan 'stats_validation'
    ROW                                                                     COLUMN+CELL                                                                                                                                                                                                      
    
    ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
            at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    hbase(main):035:0> scan '.META.'
    ROW                                                                     COLUMN+CELL                                                                                                                                                                                                      
    
    ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
            at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769)
            at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

     执行hbase hbck,发现也是-ROOT-,,0 offline

    $ hbase hbck
    
    Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
    Mon Apr 29 21:50:04 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:05 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:06 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:07 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:09 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:11 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:15 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:19 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:27 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    Mon Apr 29 21:50:43 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
    
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:200)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:127)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127)
        at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:1202)
        at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:169)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:1727)

     其中'.META.'和'-ROOT-'表的原理和关系可以参考

    https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/

    修复过程

    停止hbase集群

    $ stop-hbase.sh

    执行hbase元数据修复,报错可以先不用管

    $ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
    
    RROR: Unable to load region info for table vendor_source_rf_10066!  It may be an invalid format or version file.  You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
    19/04/29 22:18:44 ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066!  It may be an invalid format or version file.  You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
    java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo
            at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1840)
            at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1831)
            at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:578)
            at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
            at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
            at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:274)
            at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:303)
            at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:423)
            at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
    19/04/29 22:18:49 ERROR util.HBaseFsck: Bailed out due to: 
    org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066!  It may be an invalid format or version file.  You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry.
            at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:311)
            at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:423)
            at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)

    启动集群,查看master的日志输出

    Mon Apr 29 22:23:18 EDT 2019 Starting master on ec2d-cntprocnn-01
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 120522
    max locked memory       (kbytes, -l) 32
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 32768
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 120522
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    2019-04-29 22:23:20,723 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
    on.
    2019-04-29 22:23:20,791 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
    2019-04-29 22:23:21,161 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati
    on.
    2019-04-29 22:23:21,380 ERROR org.mortbay.log: Error starting handlers
    java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
            at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:443)
            at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:76)
            at org.mortbay.log.Log.warn(Log.java:175)
            at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:183)
            at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:61)
            at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
            at org.mortbay.jetty.Server.doStart(Server.java:224)
            at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
            at org.apache.hadoop.http.HttpServer.start(HttpServer.java:617)
            at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:760)
            at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:481)
            at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336)
            at java.lang.Thread.run(Thread.java:619)
    2019-04-29 22:23:27,087 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 1028785192 on ec2d-cntprocdn-02.mypna.com,60020,1556591459397 with ec2d-cntprocdn-02.mypna.com,60020,1556591459397
    2019-04-29 22:23:28,118 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
    2019-04-29 22:23:28,118 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state.
    2019-04-29 22:23:30,684 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591010682, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:31,442 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591011439, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:31,795 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591011790, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:32,138 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012131, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:32,447 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012444, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:32,471 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_poi_10066,52000100000000000001,1544522822830.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=1556591562176, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:32,754 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012748, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:33,070 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591013067, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:33,346 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591013340, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:34,575 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,04000100000000000001,1544528708735.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=1556591014574, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:34,592 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,06003300000715802137,1550466407903.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=1556591564305, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:34,602 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,07003300000751330615,1550466821719.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=1556591564386, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:34,849 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,08003300001000327984,1544543791636.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=1556591564524, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:34,857 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,09003300000715160412,1544536989039.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=1556591564612, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:35,662 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591015659, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:36,076 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,18003300001074178697,1544542488834.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=1556591565768, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:36,426 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591016424, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:36,444 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,22003300001092821096,1550456353523.3105e4d146648aff31262166669cd170. state=OPEN, ts=1556591566201, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:36,617 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,24003300000900883934,1550466042819.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=1556591566353, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:36,792 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591016789, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:37,605 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591017604, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:37,755 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,37003300001003663497,1550466126727.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=1556591567559, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:38,013 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018011, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:38,669 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018667, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:38,787 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018785, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:39,238 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019236, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:39,341 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019339, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:39,446 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019444, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:39,626 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019625, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:39,727 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019724, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,095 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020094, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,259 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020257, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,345 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020341, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,418 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020417, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,508 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020502, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,588 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020585, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,674 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but re
    gion was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020671, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,834 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020832, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,911 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020909, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:40,986 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020983, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,054 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021051, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,121 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021119, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,254 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,72003300001083634933,1544542493454.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=1556591571048, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:41,312 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021310, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,430 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021427, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,483 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021482, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,485 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,75003300001027972012,1544537176586.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=1556591571309, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:41,538 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,76000100000000000001,1544528177250.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=1556591571346, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:41,592 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021588, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:41,646 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021645, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:42,015 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,81003300000720160733,1544537024260.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=1556591571814, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:42,243 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,83003300000751674816,1550456452615.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=1556591572066, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:42,323 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,83003300001023031304,1550456452615.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=1556591572108, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:42,360 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591022358, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:42,363 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,84003300000715388172,1544531717306.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=1556591572170, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:42,367 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,84000100000000000001,1544528141770.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=1556591572183, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:23:42,509 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in  the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591022505, server=null and not in PENDING_OPEN or OPENING
    2019-04-29 22:23:42,513 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in  the state vendor_source_rf_10066,86000100000000000001,1544529154135.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=1556591572328, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states
    2019-04-29 22:30:03,217 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-04.mypna.com,60020,1556591001078 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
    org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
            at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
            at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1624)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1329)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1304)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1299)
            at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:381)
            at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:351)
            at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:637)
            at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:518)
            at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336)
            at java.lang.Thread.run(Thread.java:619)
    Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN.
            at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2479)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2437)
            at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
            at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
            at $Proxy11.openRegion(Unknown Source)
            at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:435)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1592)
            ... 9 more
    2019-04-29 22:30:03,232 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-01.mypna.com,60020,1556591551521 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
    org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
            at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
            at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1624)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1329)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1304)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1299)
            at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:381)
            at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:353)
            at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:637)
            at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:518)
            at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336)
            at java.lang.Thread.run(Thread.java:619)
    Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN.
            at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2479)
            at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2437)
            at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
            at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
    
            at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
            at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
            at $Proxy11.openRegion(Unknown Source)
            at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:435)
            at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1592)

     再次查看region信息

    hbase(main):023:0* status 'simple'
    7 live servers
        ec2d-cntprocdn-01.mypna.com:60020 1556591551521
            requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480
        ec2d-cntprocdn-04.mypna.com:60020 1556591001078
            requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480
        ec2d-cntprocdn-03.mypna.com:60020 1556591001666
            requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480
        ec2d-cntprocdn-06.mypna.com:60020 1556591062296
            requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480
        ec2d-cntprocdn-05.mypna.com:60020 1556591098649
            requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480
        ec2d-cntprocdn-02.mypna.com:60020 1556591459397
            requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480
        ec2d-cntprocdn-07.mypna.com:60020 1556591049767
            requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480
    0 dead servers
    Aggregate load: 0, regions: 442
  • 相关阅读:
    Gitlab_ansible_jenkins三剑客⑥Jenkins和ansible集成
    Gitlab_ansible_jenkins三剑客⑤jenkins Pipeline-job的使用
    Gitlab_ansible_jenkins三剑客④jenkins安装图解及freestyle的简单使用
    Gitlab_ansible_jenkins三剑客③Ansible的安装及使用
    Gitlab_ansible_jenkins三剑客②Gitlab的后台管理
    Gitlab_ansible_jenkins三剑客①搭建gitlab的详细步骤
    windows2012服务器中安装php7+mysql5.7+apache2.4环境
    07binlog日志介绍
    open()系统调用的实现
    文件系统中的目录查找
  • 原文地址:https://www.cnblogs.com/ilifeilong/p/10794844.html
Copyright © 2020-2023  润新知