• YARN executor user



    The Shell action has the following caveats:

    • Interactive commands are not supported.
    • MR1:
      • In an unsecure cluster, everything is run as the user who started the TaskTracker where our shell script is running (mapred user); in a “Kerberized” cluster, it will run as the UNIX user of whomever submitted the workflow. This is in contrast to MapReduce-based actions, which, for the purposes of interaction with Hadoop, are run as the user who submitted the workflow –although the UNIX process for the task still runs as mapred.
    • YARN/MR2:
      • The user everything is run as depends entirely on your ContainerExecutor YARN configurations (i.e.yarn.nodemanager.container-executor.class). If the DefaultContainerExecutor is being used, then everything will run as the user who started the NodeManagers (yarn user). If LinuxContainerExecutor is being used, then everything will run as whoever is configured in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (“nobody” by default). If LinuxContainerExecutor is being used and yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-user is set to “true”, then everything will run as the user who submitted the job. Alternatively, in a Kerberized cluster with LinuxContainerExecutor, everything will always run as the user who submitted the job. This can be a bit confusing, so I created this table:

    • The Shell action is executed on an arbitrary node in the cluster.
    • Different operating systems may have different versions of the same shell commands.

    The implications of that third caveat are very important. Oozie executes the shell action in the same way it executes any of the other actions: as a MapReduce job. In this case, it’s a 1-mapper-0-reducer job, which is why it can be executed on any node in the cluster. This means that any command or script that we want to execute has to be available on that node; because we don’t know which node the shell action will be executed on, the command or script has to be available on all nodes! This is fine for typical built-in shell commands like echo or grep, but can be more problematic for programs such as matlab, which must not only be installed but may also require a license. Instead, we’ll be putting our script in the same directory as the workflow.xml and taking advantage of the<file>tag to have Oozie copy it to the proper node for us.

    Even though two operating systems, or even two different versions of the same operating system, may have the same built-in commands or programs, they may behave differently or accept different arguments. For example, we’ll be using the tail command later; on Mac OS 10.7.5 we can specify the number of lines with the following arguments, but this won’t work properly on CentOS:


    more:

    http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/


    正因为当初对未来做了太多的憧憬,所以对现在的自己尤其失望。生命中曾经有过的所有灿烂,终究都需要用寂寞来偿还。
  • 相关阅读:
    保证在浏览器上word/图片/Excel的下载的表现形式一样
    C学习笔记 知识集锦(二)
    函数返回值
    C++ 四种新式类型转换
    构造&析构
    指针与引用
    C++三大特性 封装 继承 多态
    Git stash 常见用法
    Git branch && Git checkout常见用法
    linux Vi使用
  • 原文地址:https://www.cnblogs.com/candlia/p/11920253.html
Copyright © 2020-2023  润新知