• YARN executor user



    The Shell action has the following caveats:

    • Interactive commands are not supported.
    • MR1:
      • In an unsecure cluster, everything is run as the user who started the TaskTracker where our shell script is running (mapred user); in a “Kerberized” cluster, it will run as the UNIX user of whomever submitted the workflow. This is in contrast to MapReduce-based actions, which, for the purposes of interaction with Hadoop, are run as the user who submitted the workflow –although the UNIX process for the task still runs as mapred.
    • YARN/MR2:
      • The user everything is run as depends entirely on your ContainerExecutor YARN configurations (i.e.yarn.nodemanager.container-executor.class). If the DefaultContainerExecutor is being used, then everything will run as the user who started the NodeManagers (yarn user). If LinuxContainerExecutor is being used, then everything will run as whoever is configured in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (“nobody” by default). If LinuxContainerExecutor is being used and yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-user is set to “true”, then everything will run as the user who submitted the job. Alternatively, in a Kerberized cluster with LinuxContainerExecutor, everything will always run as the user who submitted the job. This can be a bit confusing, so I created this table:

    • The Shell action is executed on an arbitrary node in the cluster.
    • Different operating systems may have different versions of the same shell commands.

    The implications of that third caveat are very important. Oozie executes the shell action in the same way it executes any of the other actions: as a MapReduce job. In this case, it’s a 1-mapper-0-reducer job, which is why it can be executed on any node in the cluster. This means that any command or script that we want to execute has to be available on that node; because we don’t know which node the shell action will be executed on, the command or script has to be available on all nodes! This is fine for typical built-in shell commands like echo or grep, but can be more problematic for programs such as matlab, which must not only be installed but may also require a license. Instead, we’ll be putting our script in the same directory as the workflow.xml and taking advantage of the<file>tag to have Oozie copy it to the proper node for us.

    Even though two operating systems, or even two different versions of the same operating system, may have the same built-in commands or programs, they may behave differently or accept different arguments. For example, we’ll be using the tail command later; on Mac OS 10.7.5 we can specify the number of lines with the following arguments, but this won’t work properly on CentOS:


    more:

    http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/


    正因为当初对未来做了太多的憧憬,所以对现在的自己尤其失望。生命中曾经有过的所有灿烂,终究都需要用寂寞来偿还。
  • 相关阅读:
    numpy学习(将条件逻辑表述为数组运算)
    numpy学习(利用数组进行数据处理)
    numpy学习(通用函数:快速的元素级数组函数)
    numpy学习(数组转置和轴对换)
    numpy学习(花式索引)
    关于C++中的虚拟继承的一些总结
    adb常用命令
    进程隐藏的方法
    Microsoft Detours 2.1简介
    ebay如何确定同一电脑登陆了多个账号,以及同一账号登陆过多台电脑
  • 原文地址:https://www.cnblogs.com/candlia/p/11920253.html
Copyright © 2020-2023  润新知