[RM HA4] RM状态存储与还原原理详解

 

RM状态存储与还原机制详解

转载请注明原始链接http://www.cnblogs.com/shenh062326/p/3562199.html。

摘要

　　本文基于Apache Hadoop社区最新release版本2.3.0编写，社区从2.3.0开始支持ResourceManager的HA，已经实现的功能包括RM状态信息的存储与还原，客户端（clientToRM, AMToRM, NMToRM）实现RPC重试机制，增加Active与Standby RM角色，支持手动切换Active与Standby RM，还不支持RM的热备（自动主备切换），不支持RM切换后正在运行运行的Container不被Kill。本文将从代码上详细分析RM状态存储与还原的原理。

RM为什么要存储状态信息，RM要存储什么状态信息

　　RM HA这项工作的目的是，当Active RM由于异常无法工作时，Standby RM能接替正在服务的Active RM，防止集群出现不可用状态。那Standby RM是如何知道正在运行的App的状态的呢？

　　这就需要主Active RM把App的状态保存下来，那要保存哪些状态呢？除了保存App的状态，是否还需要保存NodeManager的状态呢？

　　我们先从代码上来看，RM HA的org.apache.hadoop.yarn.server.resourcemanager.recovery包中，定义了一个RMStateStore接口，接口解释如下：

Base class to implement storage of ResourceManager state. Takes care of asynchronous notifications and interfacing with YARN objects. Real store implementations need to derive from it and implement blocking store and load methods to actually store and load the state.

大致意思是，RMStateStore是存储ResourceManager状态的基础接口，真实的存储器需要实现存储和加载方法。

　　RMStateStore的类图如图1所示：

图1，RMStateStore的类图

　　其中包含四个内部RMState，ApplicationState，ApplicationAttemptState，RMDTSecretManager，另外还包括存储，加载，删除ApplicationState/ ApplicationAttemptState/ RMDTSecretManager的方法。

这里解释一下四个内部类，RMState是ResourceManager的状态，它包含：

Map<ApplicationId, ApplicationState> appState

RMDTSecretManagerState rmSecretManagerState

ApplicationState是Application的状态，即客户端提交的作业的状态，它包含：

final ApplicationSubmissionContext context;

final long submitTime;

final String user;

Map<ApplicationAttemptId, ApplicationAttemptState> attempts

一个ApplicationState可能包含一到多个ApplicationAttemptState，客户端提交的作业可以设置ApplicationMaster重试的次数，例如客户端设置重试两次，第一次失败，第二次成功，则ApplicationState会包含两个ApplicationAttemptState。它包含：

final ApplicationAttemptId attemptId;

final Container masterContainer;

final Credentials appAttemptCredentials;

RMDTSecretManager包含：

Map<RMDelegationTokenIdentifier, Long> delegationTokenState

Set<DelegationKey> masterKeyState

int dtSequenceNumber

　　上述就是RM存储的状态信息。当然，RMStateStore还维护着版本信息，因为存储的内容可能随着后续版本的更新而改变，这时就需要通过增加版本号来区分，这里规定了高版本需要兼容低版本的状态信息。

RM什么时候存储状态

　　顺着RMStateStore接口的六个存储方法（storeNewApplication/updateApplicationState/storeNewApplicationAttempt/updateApplicationAttemptState/storeRMDelegationTokenAndSequenceNumber/storeRMDTMasterKey）向上找，我们就能找到存储与更新RM状态的地方。

一．storeNewApplication

　　首先是storeNewApplication，顺着代码，我们很快就能发现，是在客户端提交一个作业到RM后，RM的ClientRMService.submitApplication()方法调用rmAppManager.submitApplication()方法，这里会生成一个RMAppEvent的START事件，并交给RM的异步处理器：

rmContext.getDispatcher().getEventHandler()

    .handle(new RMAppEvent(applicationId,

    isRecovered ? RMAppEventType.RECOVER : RMAppEventType.START)

RMAppImpl定义了RMApp的状态机，

        .addTransition(RMAppState.NEW, RMAppState.NEW_SAVING,

        　　RMAppEventType.START, new RMAppNewlySavingTransition())

　　其中NEW状态遇到START事件将调用RMAppNewlySavingTransition转换，它调用了app.rmContext.getStateStore().storeNewApplication(app);这样在作业刚提交的时候RM就把App存储下来了。

二．updateApplicationState

    再来看updateApplicationState，即更新App状态，从调用关系（图2）可以看到，RM通过AttemptFailedTransition和FinalSavingTransition方法调用了更新App的操作。

图2 updateApplicationState调用关系图

从RMApp的状态机可以看到，有以下地方调用FinalSavingTransition，

.addTransition(RMAppState.NEW_SAVING, RMAppState.FINAL_SAVING,

　　RMAppEventType.KILL,

　　new FinalSavingTransition(

　　new AppKilledTransition(), RMAppState.KILLED))

.addTransition(RMAppState.NEW_SAVING, RMAppState.FINAL_SAVING,

　　RMAppEventType.APP_REJECTED,

　　new FinalSavingTransition(new AppRejectedTransition(),

　　RMAppState.FAILED))

.addTransition(RMAppState.SUBMITTED, RMAppState.FINAL_SAVING,

　　RMAppEventType.APP_REJECTED,

　　new FinalSavingTransition(

　　new AppRejectedTransition(), RMAppState.FAILED))

.addTransition(RMAppState.RUNNING, RMAppState.FINAL_SAVING,

　　RMAppEventType.ATTEMPT_UNREGISTERED,
　　
　　new FinalSavingTransition(

　　new AttemptUnregisteredTransition(),

　　RMAppState.FINISHING, RMAppState.FINISHED))

.addTransition(RMAppState.KILLING, RMAppState.FINAL_SAVING,

　　RMAppEventType.ATTEMPT_KILLED,

　　new FinalSavingTransition(

　　new AppKilledTransition(), RMAppState.KILLED))

　　这里解释一下第一个状态转换原理，其他状态转换可以用相同的方法理解。App在保存后，到达RMAppState.NEW_SAVING状态，这时，如果遇到RMAppEventType.KILL事件，则会调用FinalSavingTransition方法，最后会更新已经存储的App的状态（包括结束时间和状态）。

　　调用AttemptFailedTransition状态转换的地方包括以下两个地方：

.addTransition(RMAppState.ACCEPTED,

　　EnumSet.of(RMAppState.SUBMITTED, RMAppState.FINAL_SAVING),

　　RMAppEventType.ATTEMPT_FAILED,

　　new AttemptFailedTransition(RMAppState.SUBMITTED))

.addTransition(RMAppState.RUNNING,

　　EnumSet.of(RMAppState.SUBMITTED, RMAppState.FINAL_SAVING),

　　RMAppEventType.ATTEMPT_FAILED,

　　new AttemptFailedTransition(RMAppState.SUBMITTED))

　　分别是RMAppState.ACCEPTED和RMAppState.RUNNING时遇到RMAppEventType.ATTEMPT_FAILED事件，这时候就会通过AttemptFailedTransition更新App的状态。

三 storeNewApplicationAttempt

　　什么时候存储ApplicationAttempt呢，图3 展示了storeNewApplicationAttempt的调用栈，两个调用的地方分别是AMContainerAllocatedTransition和ScheduleTransition。

图3 storeNewApplicationAttempt的调用关系

　　从RMAppAttemptImpl的状态机可以看到，RMAppAttemptState.SCHEDULED状态遇到RMAppAttemptEventType.CONTAINER_ALLOCATED事件时调用AMContainerAllocatedTransition，从而存储ApplicationAttempt状态。

.addTransition(RMAppAttemptState.SCHEDULED,

　　RMAppAttemptState.ALLOCATED_SAVING,

　　RMAppAttemptEventType.CONTAINER_ALLOCATED,

　　new AMContainerAllocatedTransition())

ScheduleTransition中存储的是UnmanageredAM的Attempt。

四． updateApplicationAttemptState

    从图4可以看到updateApplicationAttemptState的调用关系，三个调用的地方分别是AMUnregisteredTransition，ContainerFinishedTransition，FinalSavingTransition。

图4 updateApplicationAttemptState的调用关系

　　从RMAppAttemptImpl的状态机可以看到，RMAppAttemptState.RUNNING遇到RMAppAttemptState.FINISHED事件时调用AMUnregisteredTransition，遇到RMAppAttemptEventType.CONTAINER_FINISHED时调用ContainerFinishedTransition。

.addTransition(RMAppAttemptState.RUNNING,

　　EnumSet.of(RMAppAttemptState.FINAL_SAVING, RMAppAttemptState.FINISHED),

　　RMAppAttemptEventType.UNREGISTERED, new AMUnregisteredTransition())

.addTransition(

　　RMAppAttemptState.RUNNING,

　　EnumSet.of(RMAppAttemptState.RUNNING, RMAppAttemptState.FINAL_SAVING),

　　RMAppAttemptEventType.CONTAINER_FINISHED,

　　new ContainerFinishedTransition())

调用FinalSavingTransition的地方比较多，如下所示，总结起来就是Attempt遇到非正常状态下的状态转换关系。

.addTransition(RMAppAttemptState.NEW, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new BaseFinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(RMAppAttemptState.NEW, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.REGISTERED,

　　new FinalSavingTransition(

　　new UnexpectedAMRegisteredTransition(), RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.SUBMITTED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.APP_REJECTED,

　　new FinalSavingTransition(new AppRejectedTransition(),

　　RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.SUBMITTED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new BaseFinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(RMAppAttemptState.SUBMITTED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.REGISTERED,

　　new FinalSavingTransition(

　　new UnexpectedAMRegisteredTransition(), RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.SCHEDULED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new BaseFinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(RMAppAttemptState.ALLOCATED_SAVING,

　　RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new BaseFinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(RMAppAttemptState.LAUNCHED_UNMANAGED_SAVING,

　　RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.REGISTERED,

　　new FinalSavingTransition(

　　new UnexpectedAMRegisteredTransition(), RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.LAUNCHED_UNMANAGED_SAVING,

　　RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new BaseFinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(RMAppAttemptState.ALLOCATED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.LAUNCH_FAILED,

　　new FinalSavingTransition(new LaunchFailedTransition(),

　　RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.ALLOCATED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(

　　new KillAllocatedAMTransition(), RMAppAttemptState.KILLED))

 

.addTransition(RMAppAttemptState.ALLOCATED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.CONTAINER_FINISHED,

　　new FinalSavingTransition(

　　new AMContainerCrashedTransition(), RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.LAUNCHED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.CONTAINER_FINISHED,

　　new FinalSavingTransition(

　　new AMContainerCrashedTransition(), RMAppAttemptState.FAILED))

.addTransition(

　　RMAppAttemptState.LAUNCHED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.EXPIRE,

　　new FinalSavingTransition(EXPIRED_TRANSITION,

　　RMAppAttemptState.FAILED))

.addTransition(RMAppAttemptState.LAUNCHED, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new FinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

.addTransition(

　　RMAppAttemptState.RUNNING, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.EXPIRE,

　　new FinalSavingTransition(EXPIRED_TRANSITION,

　　RMAppAttemptState.FAILED))

.addTransition(

　　RMAppAttemptState.RUNNING, RMAppAttemptState.FINAL_SAVING,

　　RMAppAttemptEventType.KILL,

　　new FinalSavingTransition(new FinalTransition(

　　RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))

五．storeRMDelegationTokenAndSequenceNumber

安全部分，暂时略过

六．storeRMDTMasterKey

安全部分，暂时略过

RM如何存储状态信息

RMStateStore只实现了存储状态的接口，具体的存储方法由实现类完成，RMStateStore的实现类如图5所示。

图5 RMStateStore的实现类

　　NullRMStateStore的实现都为空方法。

　　MemoryRMStateStore使用RMStats对象存储所有RM的状态，并在内存中维护。

    FileSystemRMStateStore中，由配置yarn.resourcemanager.fs.state-store.uri决定存储的位置，由代码：

fs = fsWorkingPath.getFileSystem(conf)

和图6可以看出，存储的位置可以是本地文件系统，也可以是HDFS。

图6 Hadoop文件系统的实现类

　　ZKRMStateStore，很明显，实现了把RM状态存储到ZK上，目录结构如下所示：

ROOT_DIR_PATH

* |--- VERSION_INFO

* |--- RM_ZK_FENCING_LOCK

* |--- RM_APP_ROOT

* | |----- (#ApplicationId1)

* | | |----- (#ApplicationAttemptIds)

* | |

* | |----- (#ApplicationId2)

* | | |----- (#ApplicationAttemptIds)

* | ....

* |

* |--- RM_DT_SECRET_MANAGER_ROOT

* |----- RM_DT_SEQUENTIAL_NUMBER_ZNODE_NAME

* |----- RM_DELEGATION_TOKENS_ROOT_ZNODE_NAME

* | |----- Token_1

* | |----- Token_2

* | ....

* |

* |----- RM_DT_MASTER_KEYS_ROOT_ZNODE_NAME

* | |----- Key_1

* | |----- Key_2

* ....

　　具体实现原理可以参考ZooKeeper官网http://zookeeper.apache.org，需要注意的是，在ZKRMStateStore中，所有对ZK的操作都加上了隔离机制，防止多个client对ZK同一目录的操作。

RM什么时候还原状态

    这一块比较简单，从图7可以看出，只有在RMActiveService启动的时候才会调用loadState方法，加载已经存储的RM的状态。需要指出的是，RM支持HA后，RM中的服务已经分为Always on和Active两部分，Always on的服务会在Active和Standby RM上启动，而Active服务只会在Active RM上启动，而RM启动后默认进入Standby，当前只能手动出发RM转换为Active，这时候RM就会加载已经存储的状态并还原了。

图7，loadState的调用图

RM如何还原状态

　　和"RM如何存储状态信息"相对应，还原也是通过四个实现类完成相应的加载方法：

　　NullRMStateStore的加载方法为空方法。

　　MemoryRMStateStore通过生成新的RMStats对象并把原有的状态存储到新的RMStats对象中。

　　FileSystemRMStateStore通过加载文件系统上的文件内容，还原存储状态到RMState对象中。

　　ZKRMStateStore通过加载ZK上的状态信息，还原存储状态到RMState对象中。

　　这部分比较简单，可以通过查看相应的实现方法了解具体的还原过程。

小结

　　本文通过分析RMStateStore接口，对RM HA中状态存储与还原部分做了详细的描述，读者如果有任何问题都可以留言咨询。接下来我还将分析RMApp和RMAppattempt状态转换机制，继续期待！