• Hadoop源码分析30 JobInProgress 的 TaskInProgress 执行情况


     

    提交命令行:hadoop  jar /opt/hadoop-1.0.0/hadoop-examples-1.0.0.jar wordcount /user/admin/in/yellow.txt /user/admin/out/3

    JobTracker已生成JobInProgress(job_201404200630_0001),包括以下TaskInProgress

    maps[0](task_201404200630_0001_m_000000)

    maps[1](task_201404200630_0001_m_000001)

    maps[2](task_201404200630_0001_m_000002)

    reduces[0](task_201404200630_0001_r_000000)

    reduces[1](task_201404200630_0001_r_000001)

    cleanup[0]task_201404200630_0001_m_000003mapcleanup

    cleanup[1]task_201404200630_0001_r_000002reducecleanup

    setup[0]task_201404200630_0001_m_000004mapsetup

    setup[1]task_201404200630_0001_r_000003reducesetup

     

    RPC请求:heartbeat from10.1.1.102

    处理:获得setup[0]TaskInProgressattempt_201404200630_0001_m_000004,封装成MapTaskattempt_201404200630_0001_m_000004_0),再封装成LaunchTaskAction

    返回:1LaunchTaskAction(setup[0])

     

    RPC请求:heartbeat from10.1.1.103

    处理:没有获得Task

    返回: []

     

    RPC请求:heartbeat from10.1.1.102

    处理: Task attempt_201404200630_0001_m_000004)成功——setup[0]task_201404200630_0001_m_000004成功

    获得maps[0](task_201404200630_0001_m_000000),封装成MapTaskattempt_201404200630_0001_m_000000_0,再封装成LaunchTaskAction

    获得maps[1](task_201404200630_0001_m_000001),封装成MapTaskattempt_201404200630_0001_m_000001_0,再封装成LaunchTaskAction

    获得setup[0](attempt_201404200630_0001_m_000004_0),再封装成KillTaskAction

    返回:2LaunchTaskActionmaps[0],maps[1]),1KillTaskActionsetup[0]

     

    RPC请求:heartbeat from10.1.1.103

    处理:获得maps[2](task_201404200630_0001_m_000002),封装成MapTaskattempt_201404200630_0001_m_000002_0,再封装成LaunchTaskAction

    返回: 1LaunchTaskActionmaps[2])

     

    RPC请求:heartbeat from10.1.1.102

    处理: Task attempt_201404200630_0001_m_000000_0)成功——maps[0]task_201404200630_0001_m_000009成功

    Task attempt_201404200630_0001_m_000001_0)成功——maps[1]task_201404200630_0001_m_000001成功

    获得maps[2](task_201404200630_0001_m_000002),封装成MapTaskattempt_201404200630_0001_m_000002_1),再封装成LaunchTaskAction

    获得reduces[0]task_201404200630_0001_r_000000),封装成ReduceTaskattempt_201404200630_0001_r_000000_0),再封装成LaunchTaskAction

    返回:2LaunchTaskActionmaps[2]reduces[0])

     

    RPC请求:heartbeat from10.1.1.103

    处理:Task attempt_201404200630_0001_m_000002_0)成功——maps[2]task_201404200630_0001_m_000002成功

    获得reduces[1]task_201404200630_0001_r_000001),封装成ReduceTaskattempt_201404200630_0001_r_000001_0),再封装成LaunchTaskAction

    返回:1LaunchTaskActionreduces[1])

     

    RPC请求:heartbeat from10.1.1.102

    处理:Task attempt_201404200630_0001_m_000002_1)成功——maps[2](task_201404200630_0001_m_000002)已经成功,故需要将其KILL

    Task attempt_201404200630_0001_r_000000_0 RUNNING —— reduces[0](task_201404200630_0001_r_000000)RUNNING

    返回:1KillTaskActionmaps[2]attempt_201404200630_0001_m_000002_1

     

    RPC请求:heartbeat from10.1.1.103

    处理:Task(attempt_201404200630_0001_r_000001_0COMMIT_PENDING--reduces[1]task_201404200630_0001_r_000001COMMIT_PENDING

    返回:1CommitTaskAction(reduces[1],attempt_201404200630_0001_r_000001_0)

     

    RPC请求:heartbeat from10.1.1.102

    处理:Task(attempt_201404200630_0001_r_000000_0COMMIT_PENDING--reduces[0]task_201404200630_0001_r_000000COMMIT_PENDING

    返回:1CommitTaskAction(reduces[0],attempt_201404200630_0001_r_000000_0)

     

    RPC请求:heartbeat from10.1.1.103

    处理:Task(attempt_201404200630_0001_r_000001_0SUCCEEDED--reduces[1]task_201404200630_0001_r_000001SUCCEEDED

    返回:1KillTaskAction(reduces[1],attempt_201404200630_0001_r_000001_0)

     

    RPC请求:heartbeat from10.1.1.102

    处理:Task(attempt_201404200630_0001_r_000000_0SUCCEEDED--reduces[0]task_201404200630_0001_r_000000SUCCEEDED

    返回:1LaunchTaskAction(cleanup[0],attempt_201404200630_0001_m_000003_0),1KillTaskAction(reduces[0],attempt_201404200630_0001_r_000000_0)

     

    RPC请求:heartbeat from 10.1.1.103

    处理:

    返回:[]

     

    RPC请求:heartbeat from10.1.1.102

    处理:Task(attempt_201404200630_0001_m_000003_0SUCCEEDED--cleanup[0]task_201404200630_0001_m_000003SUCCEEDED

    返回:KillJobActionKILL_JOB:job_201404200630_0001

     

    RPC请求:heartbeat from10.1.1.103

    处理:

    返回:KillJobActionKILL_JOB:job_201404200630_0001

     

    可见10.1.1.102执行了以下Task

    setup[0]MapsetupLaunch-RUNNING-SUCCEED-KILL

    maps[0]:Launch-RUNNING- SUCCEED

    maps[1]:Launch-RUNNING- SUCCEED

    maps[2]:Launch-RUNNING- SUCCEED-KILL

    reduces[0]:Launch-RUNNING-COMMIT_PENDING-COMMIT-SUCCEED-KILL

    cleanup[0]Mapcleanup:Launch-RUNNING-SUCCEED

    Job: KILL

     

    10.1.2.103执行了以下Task

    maps[2]:Launch-RUNNING- SUCCEED

    reduces[1]:Launch-RUNNING-COMMIT_PENDING-COMMIT-SUCCEED-KILL

    Job: KILL

     

    setup[1]Reducesetupcleanup[1]Reducecleanup)没有执行。

     

  • 相关阅读:
    引入C/C++动态库
    Linux新手常用命令
    使用Dotfunsctor
    C#上传数据到HTTP,HTTPS 代码示例
    C#多个泛型约束问题
    创建DataTable与DataGridView进行绑定
    递归迭代vector三种方法实现二路归并排序
    区间贪心算法
    递归和非递归实现二叉树的遍历
    C语言实现全排列和回溯法总结
  • 原文地址:https://www.cnblogs.com/leeeee/p/7276487.html
Copyright © 2020-2023  润新知