• Apache ranger整合hive报错记录


    版本信息如下:

    hadoop2.9.2

    hive 2.x

    ranger 最新版2.1.0

    在hive端部署完ranger 插件以后,在使用beeline连接查询数据库时报错,报错信息如下:

    verbose: on
    0: jdbc:hive2://192.168.0.9:10000> show databases;
    Getting log thread is interrupted, since query is done!
    Error: Error running query: java.lang.NoSuchFieldError: REPLLOAD (state=,code=0)
    org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.NoSuchFieldError: REPLLOAD
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250)
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:977)
        at org.apache.hive.beeline.Commands.execute(Commands.java:1148)
        at org.apache.hive.beeline.Commands.sql(Commands.java:1063)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1134)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:965)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:875)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:499)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:482)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    Caused by: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.NoSuchFieldError: REPLLOAD
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:218)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:269)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:324)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:460)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:447)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
        at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
        at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
        at com.sun.proxy.$Proxy52.executeStatementAsync(Unknown Source)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:294)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:497)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.NoSuchFieldError: REPLLOAD
        at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:700)
        at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:817)
        at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:604)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:472)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:329)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1158)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1145)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:184)
        ... 27 more

    出现这种错误绝大部分是由于版本冲突导致或者版本不匹配导致的,

    去ranger-admin的软件包下,查看ranger2.1.0支持的hive版本:

    我的ranger-admin目录
    #cd /data/ranger/ranger-2.1.0-SNAPSHOT-admin/ews/webapp/WEB-INF/classes/ranger-plugins/hive
     查看hive包信息:
    #ll
    total 41504
    -rw-r--r-- 1 ranger ranger   492033 Nov  5 15:18 hive-common-3.1.0.jar
    -rw-r--r-- 1 ranger ranger 40603464 Nov  5 15:20 hive-exec-3.1.0.jar
    -rw-r--r-- 1 ranger ranger   125261 Nov  5 15:20 hive-jdbc-3.1.0.jar
    -rw-r--r-- 1 ranger ranger    36852 Nov  5 15:19 hive-metastore-3.1.0.jar
    -rw-r--r-- 1 ranger ranger   566570 Nov  5 15:19 hive-service-3.1.0.jar
    -rw-r--r-- 1 ranger ranger   313702 Nov  5 15:19 libfb303-0.9.3.jar
    -rw-r--r-- 1 ranger ranger   246445 Nov  5 15:57 libthrift-0.12.0.jar
    -rw-r--r-- 1 ranger ranger    99652 Nov  5 17:18 ranger-hive-plugin-2.1.0-SNAPSHOT.jar 

    发现支持的hive版本是3.x,而我当前使用的版本是2.x,导致使用beeline出错,

    解决办法1:升级hive的版本到3.x,

    [hduser@yjt ~]$ beeline -u jdbc:hive2://192.168.0.230:10000 -n hduser
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/data1/hadoop/hive_3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/data1/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Connecting to jdbc:hive2://192.168.0.230:10000
    Connected to: Apache Hive (version 3.1.2)
    Driver: Hive JDBC (version 3.1.2)
    Transaction isolation: TRANSACTION_REPEATABLE_READ
    Beeline version 3.1.2 by Apache Hive
    0: jdbc:hive2://192.168.0.230:10000> show databases;
    INFO  : Compiling command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad): show databases
    INFO  : Concurrency mode is disabled, not creating a lock manager
    INFO  : Semantic Analysis Completed (retrial = false)
    INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
    INFO  : Completed compiling command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad); Time taken: 1.348 seconds
    INFO  : Concurrency mode is disabled, not creating a lock manager
    INFO  : Executing command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad): show databases
    INFO  : Starting task [Stage-0:DDL] in serial mode
    INFO  : Completed executing command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad); Time taken: 0.039 seconds
    INFO  : OK
    INFO  : Concurrency mode is disabled, not creating a lock manager
    +----------------+
    | database_name  |
    +----------------+
    | default        |
    +----------------+
    1 row selected (1.878 seconds)
    0: jdbc:hive2://192.168.0.230:10000> !q
    Closing: 0: jdbc:hive2://192.168.0.230:10000
    [hduser@yjt ~]$ hive --version
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/data1/hadoop/hive_3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/data1/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    Hive 3.1.2
    Git git://HW13934/Users/gates/tmp/hive-branch-3.1/hive -r 8190d2be7b7165effa62bd21b7d60ef81fb0e4af
    Compiled by gates on Thu Aug 22 15:01:18 PDT 2019
    From source with checksum 0492c08f784b188c349f6afb1d8d9847
    [hduser@yjt ~]$ 

    解决办法2:把ranger-admin hive下的包删除,然后从hive客户端拷贝相应的包到该目录下(我没有测试)

    借鉴:

    https://m.cppentry.com/bencandy.php?fid=117&id=186236

  • 相关阅读:
    Codeforces Round #733
    [ZJOI2007] 时态同步(树形dp)
    最大子树和(树形dp)
    P2015 二叉苹果树
    没有上司的舞会(经典树形dp)
    P3884 [JLOI2009]二叉树问题(LCA)
    Bin Packing Problem(线段树 + multiset)
    P4281 [AHOI2008]紧急集合 / 聚会(最近公共祖先)
    P3128 [USACO15DEC]Max Flow P(LCA 树上差分)
    java中Set接口用法
  • 原文地址:https://www.cnblogs.com/yjt1993/p/11823101.html
Copyright © 2020-2023  润新知