hadoop运行原理之Job运行(三) TaskTracker的启动及初始化

　　与JobTracker一样，TaskTracker也有main()方法，然后以线程的方式启动(继承了Runnable接口)。main()方法中主要包含两步：一是创建一个TaskTracker对象；二是启动TaskTracker线程。

public static void main(String argv[]) throws Exception {
    ...
    try {
      JobConf conf=new JobConf();
      ...
      TaskTracker tt = new TaskTracker(conf);
      ...
      tt.run();
    } catch (Throwable e) {
     ...
    }
  }

　　一、TaskTracker tt = new TaskTracker(conf);  该方法会首先获取Map和Reduce的slot数量、磁盘检查时间间隔和JobTracker的地址；然后构造一个HttpServer(Jetty服务器)对象，这样可以通过web查看该tasktracker的运行情况及所有task；接着构造TaskController对象，用来控制任务的初始化、终结和清理工作；再接着调用initialize()方法来初始化一些变量并构造TaskTracker对象；最后启动HttpServe服务器。

　　着重说下initialize()方法。这个方法真正完成了构造TaskTracker对象的工作，并初始化了一些变量。用这个单独的方法来构造TaskTracker，可以使我们多次调用它。首先initialize()方法会清理一些历史文件(第一次启动TaskTracker不会有历史文件)，并新建一些目录；接着初始化一些变量；构造JvmManager对象用来管理JVM；初始化并启动taskReportServer，Task通过RPC向该Server汇报进度；初始化distributedCacheManager对象用来管理分布式缓存；初始化InterTrackerProtocol对象，TaskTracker由此与JobTracker通信；构造两个TaskLauncher线程：mapLauncher和reduceLauncher并启动，这个线程的run函数就是不断监测taskToLaunch队列中是否有新的 TaskTracker.TaskInProgress对象加入。如果有则从中取出一个对象，然后调用TaskTracker类的 startNewTask(TaskInProgress tip)来启动一个task。

　　二、tt.run();  该方法维护一个与JobTracker的通信连接，以周期性地向JobTracker发送心跳并领取新任务。

public void run() {
    try {
      getUserLogManager().start();
      startCleanupThreads();
      boolean denied = false;
      while (running && !shuttingDown) {
        boolean staleState = false;
        try {
          // This while-loop attempts reconnects if we get network errors
          while (running && !staleState && !shuttingDown && !denied) {    //该循环条件表示与JobTracker连接成功；
              //否则的话会在外层循环中一直尝试执行该内层循环
            try {
              //如果连接JobTracker服务成功，TaskTracker就会调用offerService()函数进入主执行循环中。
              //offerService()中的循环会每隔一段时间与JobTracker通讯一次，调用transmitHeartBeat()，获得HeartbeatResponse信息。
              State osState = offerService();
              if (osState == State.STALE) {
                staleState = true;
              } else if (osState == State.DENIED) {
                denied = true;
              }
            } catch (Exception ex) {
              if (!shuttingDown) {
                LOG.info("Lost connection to JobTracker [" +
                         jobTrackAddr + "].  Retrying...", ex);
                try {
                  Thread.sleep(5000);
                } catch (InterruptedException ie) {
                }
              }
            }
          }
        } finally {
          // If denied we'll close via shutdown below. We should close
          // here even if shuttingDown as shuttingDown can be set even
          // if shutdown is not called.
          if (!denied) {
            close();
          }
        }
        if (shuttingDown) { return; }
        if (denied) { break; }
        LOG.warn("Reinitializing local state");
        initialize();
      }
      if (denied) {
        shutdown();    //关闭该TaskTracker
      }
    } catch (IOException iex) {
      LOG.error("Got fatal exception while reinitializing TaskTracker: " +
                StringUtils.stringifyException(iex));
      return;
    } catch (InterruptedException i) {
      LOG.error("Got interrupted while reinitializing TaskTracker: " +
          i.getMessage());
      return;
    }
  }

　　该方法主体是两层while循环。如果与JobTracker的连接一旦断开，则反复尝试运行内层循环，直到与JobTracker成功连接为止；连接成功后会调用内层循环中的offerService()方法与JobTracker进行通信。

State offerService() throws Exception {
    long lastHeartbeat = System.currentTimeMillis();    //上次心跳时间

    while (running && !shuttingDown) {
      try {
        long now = System.currentTimeMillis();
        
        // accelerate to account for multiple finished tasks up-front
        //计算上次心跳到现在的时间隔
        //通过完成的任务数动态调整时间间隔
        long remaining = 
          (lastHeartbeat + getHeartbeatInterval(finishedCount.get())) - now;
        
        if (remaining <= 0) {    //达到心跳时间间隔
          finishedCount.set(0);
        }
          
        while (remaining > 0) {    //未达到心跳时间间隔
          // sleeps for the wait time or 
          // until there are *enough* empty slots to schedule tasks
          synchronized (finishedCount) {
            finishedCount.wait(remaining);    //wait相差的时间(remaining)
            
            // Recompute
            now = System.currentTimeMillis();
            remaining = 
              (lastHeartbeat + getHeartbeatInterval(finishedCount.get())) - now;
            
            if (remaining <= 0) {
              // Reset count 
              finishedCount.set(0);
              break;
            }
          }
        }

        // If the TaskTracker is just starting up:
        // 1. Verify the versions matches with the JobTracker
        // 2. Get the system directory & filesystem
        if(justInited) {
          String jtBuildVersion = jobClient.getBuildVersion();
          String jtVersion = jobClient.getVIVersion();
          if (!isPermittedVersion(jtBuildVersion, jtVersion)) {
            String msg = "Shutting down. Incompatible version or build version." +
              "TaskTracker version '" + VersionInfo.getVersion() +
              "' and build '" + VersionInfo.getBuildVersion() +
              "' and JobTracker version '" + jtVersion +
              "' and build '" + jtBuildVersion +
              " and " + CommonConfigurationKeys.HADOOP_RELAXED_VERSION_CHECK_KEY +
              " is " + (relaxedVersionCheck ? "enabled" : "not enabled") +
              " and " + CommonConfigurationKeys.HADOOP_SKIP_VERSION_CHECK_KEY +
              " is " + (skipVersionCheck ? "enabled" : "not enabled");
            LOG.fatal(msg);
            try {
              jobClient.reportTaskTrackerError(taskTrackerName, null, msg);
            } catch(Exception e ) {
              LOG.info("Problem reporting to jobtracker: " + e);
            }
            return State.DENIED;
          }
          
          String dir = jobClient.getSystemDir();
          while (dir == null) {
            LOG.info("Failed to get system directory...");
            
            // Re-try
            try {
              // Sleep interval: 1000 ms - 5000 ms
              int sleepInterval = 1000 + r.nextInt(4000);
              Thread.sleep(sleepInterval);
            } catch (InterruptedException ie) 
            {}
            dir = jobClient.getSystemDir();
          }
          systemDirectory = new Path(dir);
          systemFS = systemDirectory.getFileSystem(fConf);
        }

        now = System.currentTimeMillis();
        if (now > (lastCheckDirsTime + diskHealthCheckInterval)) {    //是否达到磁盘检查间隔时间
          localStorage.checkDirs();
          lastCheckDirsTime = now;
          int numFailures = localStorage.numFailures();
          // Re-init the task tracker if there were any new failures
          if (numFailures > lastNumFailures) {
            lastNumFailures = numFailures;
            return State.STALE;
          }
        }

        // Send the heartbeat and process the jobtracker's directives
        //发送Heartbeat到JobTracker，得到response 
        HeartbeatResponse heartbeatResponse = transmitHeartBeat(now);

        // Note the time when the heartbeat returned, use this to decide when to send the
        // next heartbeat   
        lastHeartbeat = System.currentTimeMillis();
        
        // Check if the map-event list needs purging
        Set<JobID> jobs = heartbeatResponse.getRecoveredJobs();
        if (jobs.size() > 0) {
          synchronized (this) {
            // purge the local map events list
            for (JobID job : jobs) {
              RunningJob rjob;
              synchronized (runningJobs) {
                rjob = runningJobs.get(job);          
                if (rjob != null) {
                  synchronized (rjob) {
                    FetchStatus f = rjob.getFetchStatus();
                    if (f != null) {
                      f.reset();
                    }
                  }
                }
              }
            }

            // Mark the reducers in shuffle for rollback
            synchronized (shouldReset) {
              for (Map.Entry<TaskAttemptID, TaskInProgress> entry 
                   : runningTasks.entrySet()) {
                if (entry.getValue().getStatus().getPhase() == Phase.SHUFFLE) {
                  this.shouldReset.add(entry.getKey());
                }
              }
            }
          }
        }
        
        //调用HeartbeatResponse的getActions()函数获得JobTracker传过来的所有指令，即一个TaskTrackerAction数组
        TaskTrackerAction[] actions = heartbeatResponse.getActions();
        if(LOG.isDebugEnabled()) {
          LOG.debug("Got heartbeatResponse from JobTracker with responseId: " + 
                    heartbeatResponse.getResponseId() + " and " + 
                    ((actions != null) ? actions.length : 0) + " actions");
        }
        if (reinitTaskTracker(actions)) {
          return State.STALE;
        }
            
        // resetting heartbeat interval from the response.
        heartbeatInterval = heartbeatResponse.getHeartbeatInterval();
        justStarted = false;
        justInited = false;
        if (actions != null){ 
        //遍历这个数组
          for(TaskTrackerAction action: actions) {
            if (action instanceof LaunchTaskAction) {    //如果是LaunchTaskAction，则调用调用addToTaskQueue加入到待执行队列
              addToTaskQueue((LaunchTaskAction)action);
            } else if (action instanceof CommitTaskAction) {    //如果是CommitTaskAction，则添加到需要提交的队列中
              CommitTaskAction commitAction = (CommitTaskAction)action;
              if (!commitResponses.contains(commitAction.getTaskID())) {
                LOG.info("Received commit task action for " + 
                          commitAction.getTaskID());
                commitResponses.add(commitAction.getTaskID());
              }
            } else {    //如果是KillJobAction或者KillTaskAction，则加入到tasksToCleanup队列，交给一个taskCleanupThread线程来处理
              addActionToCleanup(action);
            }
          }
        }
        //杀死一定时间没没有汇报进度的task 
        markUnresponsiveTasks();
        
        //当剩余磁盘空间小于mapred.local.dir.minspacekill(默认为0)时，寻找合适的任务将其杀掉以释放空间
        killOverflowingTasks();
            
        //we've cleaned up, resume normal operation
        if (!acceptNewTasks && isIdle()) {
          acceptNewTasks=true;
        }
        //The check below may not be required every iteration but we are 
        //erring on the side of caution here. We have seen many cases where
        //the call to jetty's getLocalPort() returns different values at 
        //different times. Being a real paranoid here.
        checkJettyPort(server.getPort());
      } catch (InterruptedException ie) {
        LOG.info("Interrupted. Closing down.");
        return State.INTERRUPTED;
      } catch (DiskErrorException de) {
        String msg = "Exiting task tracker for disk error:\n" +
          StringUtils.stringifyException(de);
        LOG.error(msg);
        synchronized (this) {
          jobClient.reportTaskTrackerError(taskTrackerName, 
                                           "DiskErrorException", msg);
        }
        // If we caught a DEE here we have no good dirs, therefore shutdown.
        return State.DENIED;
      } catch (RemoteException re) {
        String reClass = re.getClassName();
        if (DisallowedTaskTrackerException.class.getName().equals(reClass)) {
          LOG.info("Tasktracker disallowed by JobTracker.");
          return State.DENIED;
        }
      } catch (Exception except) {
        String msg = "Caught exception: " + 
          StringUtils.stringifyException(except);
        LOG.error(msg);
      }
    }

    return State.NORMAL;
  }

　　通过下面这个图可以很清楚的了解这个方法的功能：

 

　　该方法的主体是一个while循环，首先检查是否达到心跳间隔时间，如果没有的话则阻塞一定时间(wait)；然后检查TaskTracker是否刚启动，如果是的话需要先检查JobTracker版本和TaskTracker版本是否一致，不一致的话返回State.DENIED；再获取system directory和filesystem；接着如果达到磁盘检查时间，则检查磁盘是否有损坏；再接着transmitHeartBeat(now)方法将当前结点的运行时信息(如资源使用情况、任务运行状态等)，通过心跳汇报给JobTracker，同时接收来自JobTracker的各种命令；JobTracker将心跳应答封装在HeartbeatResponse对象中，该对象包括两部分内容：第一是作业集合recoveredJobs，它是上次关闭JobTracker时正在运行的作业集合，重启JobTracker后需要恢复这些作业的运行状态；第二是需要执行的命令列表actions。接着会检查这些命令里是否包括了重新初始化命令(ReinitTrackerAction)，如果有则返回State.STALE，会重新初始化TaskTracker；修正心跳间隔；遍历actions数组，如果是LaunchTaskAction，则根据是MapTask还是ReduceTask，分别放进相应的mapLauncher和reduceLauncher中，这两个launcher都是TaskLauncher；如果是CommitTaskAction则加入到commitResponses；其他的类型放入tasksToCleanup.put(action)，表示要清理，如执行KillJobAction或者KillTaskAction等。kUnresponsiveTasks()方法杀死一定时间内没有汇报进度的任务；killOverflowingTasks()中，当剩余磁盘空间小于mapred.local.dir.minspacekill(默认为0)时，寻找合适的任务将其杀掉以释放空间(优先杀掉Reduce任务)。

　　下面分析下transmitHeartBeat(now)方法，这是offerService()方法中最重要的步骤。还是先来看个图，这样能对这个方法的流程有个大概了解。

　　transmitHeartBeat(now)方法主要是构造心跳信息(会把当前结点信息和该结点上的所有task信息告知JobTracker)并向JobTracker发送心跳的。首先它会检查是否达到计数器时间间隔(60s)，如果没有达到的话，那么就不发送任务的计数器(Counter)信息，否则要发送。然后检查上次心跳是否结束，如果没有结束，则继续使用上次的结点状态信息；如果结束的话则构造新的结点状态信息(会把当前结点的健康状态、资源情况以及task信息封装到里边)。接着判断该TaskTracker是否能够接收新任务，如果能够接收新任务，则将当前结点的起源情况设置到status对象里。接着再设置结点的健康状态。结点的健康状态由NodeHealthCheckerService线程来监测，该线程运行一个脚本来监控结点的运行情况；如果该线程监测到脚本输出里有“ERROR”开头行，则将结点标记为“unhealthy”；如果没有设置该脚本时，则不启动该线程，结点默认“healthy”；最后通过RPC调用JobTracker的heartbeat()方法。在得到心跳响应后，将status标记为null，表示本次心跳通信结束，并返回心跳响应。

　　关于TaskTracker的一个思考。TaskTracker与JobTracker不同的是，TaskTracker是以线程的方式启动的，而JobTracker却不是。这是否也说明了Hadoop采用的pull的模式，从这儿看出TaskTracker主动与JobTracker进行通信？

　　本文基于hadoop1.2.1

　　如有错误，还请指正

　　参考文章：《Hadoop技术内幕 深入理解MapReduce架构设计与实现原理》 董西成

　　转载请注明出处：http://www.cnblogs.com/gwgyk/p/3999811.html