Mac 环境 Hadoop 提交 job 报错 /bin/bash: /bin/java: No such file or directory

Mac 环境 Hadoop 提交 job 报错 /bin/bash: /bin/java: No such file or directory

➜  hadoop-2.9.2 bin/hadoop jar /Users/leone/Documents/hadoop-wc.jar com.leone.bigdata.hadoop.mr.wc.MrHelloWord /data/words.log /output /Users/leone/Documents/hadoop-wc.jar
20/06/20 20:55:22 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
20/06/20 20:55:23 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
20/06/20 20:55:23 INFO input.FileInputFormat: Total input files to process : 1
20/06/20 20:55:23 INFO mapreduce.JobSubmitter: number of splits:1
20/06/20 20:55:23 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/06/20 20:55:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1592657698633_0001
20/06/20 20:55:24 INFO impl.YarnClientImpl: Submitted application application_1592657698633_0001
20/06/20 20:55:24 INFO mapreduce.Job: The url to track the job: http://MacBookPro:8088/proxy/application_1592657698633_0001/
20/06/20 20:55:24 INFO mapreduce.Job: Running job: job_1592657698633_0001
20/06/20 20:55:27 INFO mapreduce.Job: Job job_1592657698633_0001 running in uber mode : false
20/06/20 20:55:27 INFO mapreduce.Job:  map 0% reduce 0%
20/06/20 20:55:27 INFO mapreduce.Job: Job job_1592657698633_0001 failed with state FAILED due to: Application application_1592657698633_0001 failed 2 times due to AM Container for appattempt_1592657698633_0001_000002 exited with  exitCode: 127
Failing this attempt.Diagnostics: [2020-06-20 20:55:27.116]Exception from container-launch.
Container id: container_1592657698633_0001_02_000001
Exit code: 127

[2020-06-20 20:55:27.118]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


[2020-06-20 20:55:27.118]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


For more detailed output, check the application tracking page: http://MacBookPro:8088/cluster/app/application_1592657698633_0001 Then click on links to logs of each attempt.
. Failing the application.
20/06/20 20:55:27 INFO mapreduce.Job: Counters: 0
➜  hadoop-2.9.2

解决方案

1. 建立 Java 软连接

sudo ln -s /usr/bin/java /bin/java

Mac下输入此命令会提示ln: /bin/java: Operation not permitted 此时需要禁用 Mac 的 SIP 特性，SIP特性大概是指即使拥有sudo权限也无法修改系统级目录的权限，默认为启用状态，至于如何禁用可以自行Google。

2. 修改 $HADOOP_HOME/ibexec/hadoop-config.sh（推荐）

if [ -x /usr/libexec/java_home ]; then
    export JAVA_HOME=($(/usr/libexec/java_home))
else
    export JAVA_HOME=(/Library/Java/Home)
fi

改为

if [ -x /usr/libexec/java_home ]; then
    // note that the extra parentheses are removed
    export JAVA_HOME=$(/usr/libexec/java_home)
else
    export JAVA_HOME=/Library/Java/Home
fi

去掉了最外围的括号（重启Hadoop）