背景
在hive sql中执行添加临时udf的操作
add jar hdfs:///user/hive/lib/tools-1.0.jar;
create temporary function decode as 'com.test.etl.Decoder';
在定时任务重偶尔会报错，报错比较随机，报错信息如下：
INFO : Query ID = hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks not specified. Estimated from input data size: 197
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=
INFO : Cleaning up the staging area /user/yarn/hive/.staging/job_1629262398375_145140
ERROR : Job Submission failed with exception 'java.io.FileNotFoundException(File file:/disk1/hive/download/tools-1.0.jar does not exist)'
java.io.FileNotFoundException: File file:/disk1/hive/download/tools-1.0.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:378)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329)
at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:680)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:313)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:205)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:441)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:146)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2250)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1893)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1327)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not exist
INFO : Completed executing command(queryId=hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6); Time taken: 0.133 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not exist (state=08S01,code=1)
Closing: 0: jdbc:hive2://master:10000
Process exited with an error: 1 (Exit value: 1)
定位
报错是偶发的，没有规律，查看报错的目录，为hive下载目录，配置为：
hive.downloaded.resources.dir=/disk1/hive/download
检查发现集群中很多节点该目录不存在，怀疑是服务器环境初始化有问题，有的节点没有正确初始化。
进一步检查发现同一个节点这个目录，有时存在，有时不存在，排除上边的怀疑。
hive中这个配置的默认值为：
hive.downloaded.resources.dir=${system:java.io.tmpdir}/${hive.session.id}_resources
可见这个目录与session相关，每次session关闭时会做删除，如果配置的路径没有sessionid，则不同session之间会相互影响
DOWNLOADED_RESOURCES_DIR("hive.downloaded.resources.dir",
"${system:java.io.tmpdir}" + File.separator + "${hive.session.id}_resources",
"Temporary local directory for added resources in the remote file system."),
修改配置为：
hive.downloaded.resources.dir=/disk1/hive/download/${hive.session.id}_resources
问题解决。