hive执行结果moveTask操作失败
Apache Hive 2.1.0 ，在执行"INSERT OVERWRITE TABLE ...... select "或者 "insert overwrite directory /tmp/data/hive-test "操作，如果生成的结果文件是多个时，执行结果文件moveTask操作会失败。最新的Apache Hive 2.1.1版本同样有该问题；Apache Hive 1.2.1版本的hive没有该问题。
具体执行的sql如下：
insert overwrite directory '/tmp/fuxin.zhao/hive-test'
select
shippingorderid
,logisticsplatformid
,stockoutorderid
,logisticstypeid
,externalshippingorderno
,packageweight
,freight
,freightstatus
,entertime
,shippingorderstatus
,shippinglog
,shippinglogupdatetime
,shippingstatustime
,confirmreceivetime
,remarks
,createtype
,lastmodifytime
,enteruser
,updatetime
,updateuser
from
(
select *,row_number() over(partition by shippingorderid order by LastModifyTime desc) as rn
from
(select * from ods.m1_shippingorder where dt = '2014-01-01'
union all
select * from fds.m1_shippingorder where dt = '2099-12-31'
) a )b where b.rn = 1
产生的异常如下：
Failed with exception org.apache.hadoop.hdfs.protocol.AclException: Invalid ACL: multiple entries with same scope, type and name.
at org.apache.hadoop.hdfs.server.namenode.AclTransformation.buildAndValidateAcl(AclTransformation.java:285)
at org.apache.hadoop.hdfs.server.namenode.AclTransformation.replaceAclEntries(AclTransformation.java:230)
at org.apache.hadoop.hdfs.server.namenode.FSDirAclOp.unprotectedSetAcl(FSDirAclOp.java:206)
at org.apache.hadoop.hdfs.server.namenode.FSDirAclOp.setAcl(FSDirAclOp.java:146)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:7938)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1813)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1330)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
异常：org.apache.hadoop.hive.ql.exec.MoveTask执行失败
##insert overwrite diretory：
2016-12-13 18:27:15,630 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 81.52 sec
MapReduce Total cumulative CPU time: 1 minutes 21 seconds 520 msec
Ended Job = job_1480497945656_0288
Moving data to directory /tmp/t_FDS/m1_shippingorder/dt=2099-12-31
Failed with exception Unable to move source hdfs://dbmtimehadoop/tmp/t_FDS/m1_shippingorder/dt=2099-12-31/.hive-staging_hive_2016-12-13_18-26-28_695_9094454822676037473-1/-ext-10000 to destination /tmp/t_FDS/m1_shippingorder/dt=2099-12-31
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://dbmtimehadoop/tmp/t_FDS/m1_shippingorder/dt=2099-12-31/.hive-staging_hive_2016-12-13_18-26-28_695_9094454822676037473-1/-ext-10000 to destination /tmp/t_FDS/m1_shippingorder/dt=2099-12-31
MapReduce Jobs Launched:
Stage-Stage-1: Map: 5 Reduce: 4 Cumulative CPU: 81.52 sec HDFS Read: 778870925 HDFS Write: 778698546 SUCCESS
Total MapReduce CPU Time Spent: 1 minutes 21 seconds 520 msec
异常：java.util.ConcurrentModificationException
Failed with exception Unable to move source hdfs://dbmtimehadoop/tmp/fuxin.zhao/hive-test/.hive-staging_hive_2016-12-22_11-45-12_256_5450334497172511865-1/-ext-10000 to destination /tmp/fuxin.zhao/hive-test
16/12/22 11:45:59 [main]: ERROR exec.Task: Failed with exception Unable to move source hdfs://dbmtimehadoop/tmp/fuxin.zhao/hive-test/.hive-staging_hive_2016-12-22_11-45-12_256_5450334497172511865-1/-ext-10000 to destination /tmp/fuxin.zhao/hive-test
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://dbmtimehadoop/tmp/fuxin.zhao/hive-test/.hive-staging_hive_2016-12-22_11-45-12_256_5450334497172511865-1/-ext-10000 to destination /tmp/fuxin.zhao/hive-test
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:103)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:255)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2984)
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFileInDfs(MoveTask.java:119)
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:96)
... 20 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at java.util.AbstractCollection.toString(AbstractCollection.java:461)
at java.lang.String.valueOf(String.java:2982)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.apache.hadoop.fs.permission.AclStatus.toString(AclStatus.java:108)
at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:75)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2961)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2953)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://dbmtimehadoop/tmp/fuxin.zhao/hive-test/.hive-staging_hive_2016-12-22_11-45-12_256_5450334497172511865-1/-ext-10000 to destination /tmp/fuxin.zhao/hive-test
16/12/22 11:45:59 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
异常：java.lang.ArrayIndexOutOfBoundsException
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2984)
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFileInDfs(MoveTask.java:119)
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:96)
... 20 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at java.util.ArrayList.removeRange(ArrayList.java:634)
at java.util.ArrayList$SubList.removeRange(ArrayList.java:1063)
at java.util.AbstractList.clear(AbstractList.java:234)
at com.google.common.collect.Iterables.removeIfFromRandomAccessList(Iterables.java:209)
at com.google.common.collect.Iterables.removeIf(Iterables.java:180)
at org.apache.hadoop.hive.io.HdfsUtils.removeBaseAclEntries(HdfsUtils.java:155)
at org.apache.hadoop.hive.io.HdfsUtils.setFullFileStatus(HdfsUtils.java:77)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2961)
at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2953)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
下面是源码中关于文件权限继承的开关代码：
HiveConf.ConfVars.HIVE_WAREHOUSE_SUBDIR_INHERIT_PERMS);
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
产生问题的原因：
hive的查询结果在在进行move操作时，需要进行文件权限的授权，多个文件的授权是并发进行的，hive中该源码是在一个线程池中
执行的，该操作在多线程时线程同步有问题的该异常，这是hive的一个bug，目前截止目前的最新版本Apache Hive 2.1.1还没有修复该问题；
可以通过关闭hive的文件权限继承 hive.warehouse.subdir.inherit.perms=false 来规避该问题。
解决方法：
hive.warehouse.subdir.inherit.perms
<property>
<name>hive.warehouse.subdir.inherit.perms</name>
<value>true</value>
<description>
Set this to false if the table directories should be created
with the permissions derived from dfs umask instead of
inheriting the permission of the warehouse or database directory.
</description>
</property>