• The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2048, vCores:2> maxContainerCapability:<memory:1024, vCores


    yarn内存设置问题

    hive查询时出现

    Ended Job = job_1544003470555_0007 with errors
    Error during job, obtaining debugging information...
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

    然后进行yarn测试

    hadoop jar hadoop-mapreduce-examples-3.0.0-cdh6.0.0.jar pi 2 10

    报错

    18/12/11 17:58:56 INFO mapreduce.Job: Job job_1544003470555_0008 failed with state KILLED due to: The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2048, vCores:2> maxContainerCapability:<memory:1024, vCores:1>

    解决方法

    修改参数 调大虚拟内存,根据自己情况配置

    mapreduce.map.memory.mb=2048
    mapreduce.reduce.memory.mb=2048
    yarn.nodemanager.vmem-pmem-ratio=3
    参考yarn平台参数设置点击此处
    这样map运行时的虚拟内存大小为 2048*3

    类似这样的情况还有
    设置Container的分配的内存大小,意味着ResourceManager只能分配给Container的内存

    大于yarn.scheduler.minimum-allocation-mb=2G,
    不能超过 yarn.scheduler.maximum-allocation-mb=8G  的值。

    ResourceManager分配给container的CPU也要满足最小和最大值的条件,通过设置

    yarn.scheduler.minimum-allocation-vcores=2
    yarn.scheduler.maximum-allocation-vcores=8

  • 相关阅读:
    压缩与解压缩 ZipHelper
    ESFramework介绍之(15)-- IRAS
    ESFramework介绍之(8)-- 客户端插件IPassiveAddin
    使用Eclipse开发Jsp
    深入探讨 java.lang.ref 包
    java多线程总结二
    模拟弹子台球java多线程应用
    简单的邮件客户端
    最近创办了一个java学习邮件列表
    优秀的XML~~Altova XMLSpy 2010英文企业版+有效破解方法
  • 原文地址:https://www.cnblogs.com/liquan-anran/p/10103893.html
Copyright © 2020-2023  润新知