The Shell action has the following caveats:
- Interactive commands are not supported.
- MR1:
- In an unsecure cluster, everything is run as the user who started the TaskTracker where our shell script is running (mapred user); in a “Kerberized” cluster, it will run as the UNIX user of whomever submitted the workflow. This is in contrast to MapReduce-based actions, which, for the purposes of interaction with Hadoop, are run as the user who submitted the workflow –although the UNIX process for the task still runs as mapred.
- YARN/MR2:
- The user everything is run as depends entirely on your
ContainerExecutor
YARN configurations (i.e.yarn.nodemanager.container-executor.class
). If theDefaultContainerExecutor
is being used, then everything will run as the user who started the NodeManagers (yarn user). IfLinuxContainerExecutor
is being used, then everything will run as whoever is configured inyarn.nodemanager.linux-container-executor.nonsecure-mode.local-user
(“nobody” by default). IfLinuxContainerExecutor
is being used andyarn.nodemanager.linux-container-executor.nonsecure-mode.limit-user
is set to “true”, then everything will run as the user who submitted the job. Alternatively, in a Kerberized cluster withLinuxContainerExecutor
, everything will always run as the user who submitted the job. This can be a bit confusing, so I created this table:
- The user everything is run as depends entirely on your
- The Shell action is executed on an arbitrary node in the cluster.
- Different operating systems may have different versions of the same shell commands.
The implications of that third caveat are very important. Oozie executes the shell action in the same way it executes any of the other actions: as a MapReduce job. In this case, it’s a 1-mapper-0-reducer job, which is why it can be executed on any node in the
cluster. This means that any command or script that we want to execute has to be available on that node; because we don’t know which node the shell action will be executed on, the command or script has to be available on all nodes! This is fine for typical
built-in shell commands like echo
or grep
,
but can be more problematic for programs such as matlab
, which must not only be installed
but may also require a license. Instead, we’ll be putting our script in the same directory as the workflow.xml and taking advantage of the<file>
tag
to have Oozie copy it to the proper node for us.
Even though two operating systems, or even two different versions of the same operating system, may have the same built-in commands or programs, they may behave differently or accept different arguments. For example, we’ll be using the tail
command
later; on Mac OS 10.7.5 we can specify the number of lines with the following arguments, but this won’t work properly on CentOS:
more:
http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/