• YARN HA Failover 导致 RM状态异常问题


    2021-11-15 18:52:15,361 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2021-11-15 18:52:15,372 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8032: starting
    2021-11-15 18:52:15,421 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8030: starting
    2021-11-15 18:52:19,190 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to active state
    2021-11-15 18:52:19,193 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
    2021-11-15 18:52:19,222 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=refreshQueues TARGET=AdminService RESULT=FAILURE DESCRIPTION=ResourceManager is not active. Can not refresh queues. PERMISSIONS=
    2021-11-15 18:52:19,222 ERROR org.apache.hadoop.yarn.server.resourcemanager.AdminService: RefreshAll failed so firing fatal event
    org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
    at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
    2021-11-15 18:52:19,223 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
    org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
    at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
    Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll during transistion to Active
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:321)
    at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
    ... 4 more
    Caused by: org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
    ... 5 more
    2021-11-15 18:52:19,223 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session
    2021-11-15 18:52:19,232 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type TRANSITION_TO_ACTIVE_FAILED. Cause:
    org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
    at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
    at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

    2021-11-15 18:52:19,234 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

    获取rm的状态

    命令:

    yarn rmadmin -getServiceState rm1

    yarn rmadmin -getServiceState rm2

    不能获取rm1,rm2的状态。

    https://issues.apache.org/jira/browse/YARN-2588
    https://issues.apache.org/jira/browse/YARN-2019
    https://issues.apache.org/jira/browse/YARN-2010
    Recovery失败导致导致RM无法启动:非常严重的bug。
    根源在于RM HA启动时会去zk读取之前的状态(recovery过程),如果zk中的数据有问题,recovery过程会抛出异常,
    这个异常没处理好,会直接抛到上层,导致RM进入STOPPED状态。
    进入STOPPED状态后,就不能再变成其他状态了。


    解决方法:

    不改代码的话,只能手动清除zk中的数据。可以手动删除zk中的/rmstore路径:
    setAcl /rmstore/ZKRMStateRoot world:anyone:cdrwa
    rmr /rmstore


    或者在yarn-site.xml中设置yarn.resourcemanager.zk-state-store.parent-path属性,比如/rmstore2,将数据存到另外一个路径。

  • 相关阅读:
    iTerm2使用技巧
    我的mac下有关php扩展的安装
    xmlhttprequest 1.0和2.0的区别,from qq前端哥
    PHP错误日志记录:display_errors与log_errors的区别
    目前php连接mysql的主要方式
    闭包介绍汇总
    接口设计知识总结
    git命令——从GitHub clone XXX分支,本地创建新分支push到远程仓库
    Spring错误——Junit测试——java.net.BindException: Address already in use: bind
    Java.util.Random生成随机数
  • 原文地址:https://www.cnblogs.com/songyuejie/p/15566475.html
Copyright © 2020-2023  润新知