hadoop 2.6 yarn Records实现分析

转自： https://blog.csdn.net/houzhizhen/article/details/51372058

Records在Yarn中原代码中主要用于RPC通訊，如以下语句生成一个新应用程序的請求，GetNewApplicationRequest request =
        Records.newRecord(GetNewApplicationRequest.class);

Records⾥的代码非常简单，调用newRecord时调用factory.newRecordInstance()方法，源代码如下：

public class Records {
  // The default record factory
  private static final RecordFactory factory =
      RecordFactoryProvider.getRecordFactory(null);
 
  public static <T> T newRecord(Class<T> cls) {
    return factory.newRecordInstance(cls);
  }
}

fractory是RecordFactory类型的，是一个接口，只有一个方法，代码如下：

public interface RecordFactory {
  public <T> T newRecordInstance(Class<T> clazz);
}

这个factory是RecordFactoryProvider.getRecordFactory(null)来初始化的。RecordFactoryProvider的源代码如下：

public class RecordFactoryProvider {
  private static Configuration defaultConf;
  
  static {
    defaultConf = new Configuration();
  }
  
  private RecordFactoryProvider() {
  }
  
  public static RecordFactory getRecordFactory(Configuration conf) {
    if (conf == null) {
      //Assuming the default configuration has the correct factories set.
      //Users can specify a particular factory by providing a configuration.
      conf = defaultConf;
    }
    String recordFactoryClassName = conf.get(
        YarnConfiguration.IPC_RECORD_FACTORY_CLASS,
        YarnConfiguration.DEFAULT_IPC_RECORD_FACTORY_CLASS);
    return (RecordFactory) getFactoryClassInstance(recordFactoryClassName);
  }
  
  private static Object getFactoryClassInstance(String factoryClassName) {
    try {
      Class<?> clazz = Class.forName(factoryClassName);
      Method method = clazz.getMethod("get", null);
      method.setAccessible(true);
      return method.invoke(null, null);
    } catch (ClassNotFoundException e) {
      throw new YarnRuntimeException(e);
    } catch (NoSuchMethodException e) {
      throw new YarnRuntimeException(e);
    } catch (InvocationTargetException e) {
      throw new YarnRuntimeException(e);
    } catch (IllegalAccessException e) {
      throw new YarnRuntimeException(e);
    }
  }
}

在方法⾥，由于传递的参数是null，所以conf = defaultConf,默认的DEFAULT_IPC_RECORD_FACTORY_CLASS 是    "org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl";调用getFactoryClassInstance("org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl")。

在getFactoryClassInstance方法⾥.，直接调用它的静态get方法，没有参数。RecordFactoryPBImpl的源代码如下： 1 public class RecordFactoryPBImpl implements Record 2

  private static final String PB_IMPL_PACKAGE_SUFFIX = "impl.pb";
  private static final String PB_IMPL_CLASS_SUFFIX = "PBImpl";
 
  private static final RecordFactoryPBImpl self = new RecordFactoryPBImpl();
  private Configuration localConf = new Configuration();
  private ConcurrentMap<Class<?>, Constructor<?>> cache = new ConcurrentHashMap<Class<?>, Constructor<?>>();
 
  private RecordFactoryPBImpl() {
  }
  
  public static RecordFactory get() {
    return self;
  }
  
  @SuppressWarnings("unchecked")
  @Override
  ....
}

RecordFactoryPBImpl是一个单例模式，get方法返回静态的final 变量 self.

在newRecordInstance(Class<T> clazz)方法⾥，首先判断cache是否能直接得到此类的构造器Constructor，如果得到，就直接调用constructor.newInstance();方法初始化一个对象，然后把此对象返回；如果缓冲器⾥没有，先调用getPBImplClassName(clazz)得到PBImpl的类实现，如getPBImplClassName(clazz)；传递的是GetNewApplicationRequest，所在包是org.apache.hadoop.yarn.api.protocolrecords，按照规則，实现类是GetNewApplicationRequestPBImpl，所在包是org.apache.hadoop.yarn.api.protocolrecords.impl.pb, getPBImplClassName()方法返回的内容是org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationRequestPBImpl。

接着调用localConf.getClassByName("org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationRequestPBImpl")来返回class对象。然后得到它的constructor对象，放到cache⾥。

public <T> T newRecordInstance(Class<T> clazz) {
    
    Constructor<?> constructor = cache.get(clazz);
    if (constructor == null) {
      Class<?> pbClazz = null;
      try {
        pbClazz = localConf.getClassByName(getPBImplClassName(clazz));
      } catch (ClassNotFoundException e) {
        throw new YarnRuntimeException("Failed to load class: ["
            + getPBImplClassName(clazz) + "]", e);
      }
      try {
        constructor = pbClazz.getConstructor();
        constructor.setAccessible(true);
        cache.putIfAbsent(clazz, constructor);
      } catch (NoSuchMethodException e) {
        throw new YarnRuntimeException("Could not find 0 argument constructor", e);
      }
    }
    try {
      Object retObject = constructor.newInstance();
      return (T)retObject;
    } catch (InvocationTargetException e) {
      throw new YarnRuntimeException(e);
    } catch (IllegalAccessException e) {
      throw new YarnRuntimeException(e);
    } catch (InstantiationException e) {
      throw new YarnRuntimeException(e);
    }
  }
 
  private String getPBImplClassName(Class<?> clazz) {
    String srcPackagePart = getPackageName(clazz);
    String srcClassName = getClassName(clazz);
    String destPackagePart = srcPackagePart + "." + PB_IMPL_PACKAGE_SUFFIX;
    String destClassPart = srcClassName + PB_IMPL_CLASS_SUFFIX;
    return destPackagePart + "." + destClassPart;
  }
  
  private String getClassName(Class<?> clazz) {
    String fqName = clazz.getName();
    return (fqName.substring(fqName.lastIndexOf(".") + 1, fqName.length()));
  }
  
  private String getPackageName(Class<?> clazz) {
    return clazz.getPackage().getName();
  }