• EMR的hive包升级和gateway启动hs2


    升级:


    mkdir /opt/apps/hive
    cd /opt/apps/hive
    wget http://emr.oss-cn-hangzhou.aliyuncs.com/ecm-repo/hive/opay/apache-hive-2.3.5-bin.tar.gz
    md5sum apache-hive-2.3.5-bin.tar.gz
    229e32d99d8d288bdece8c897d6cc625 apache-hive-2.3.5-bin.tar.gz
    tar xvf apache-hive-2.3.5-bin.tar.gz

    1. Gateway升级Hive包
    a.登录Gateway机器,将新的hive包解压到一个目录,比如/opt/apps/hive
    b. cp /usr/lib/hive-current/lib/emr-hive-hook*jar /opt/apps/hive/apache-hive-2.3.5-bin/lib
    c. unlink /usr/lib/hive-current
    d. ln -s /opt/apps/hive/apache-hive-2.3.5-bin /usr/lib/hive-current
    e. 建立其它软连接

    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-hive-plugin-impl /usr/lib/hive-current/lib/ranger-hive-plugin-impl
    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-hive-plugin-shim-1.2.0.jar /usr/lib/hive-current/lib/ranger-hive-plugin-shim-1.2.0.jar
    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-plugin-classloader-1.2.0.jar /usr/lib/hive-current/lib/ranger-plugin-classloader-1.2.0.jar

    ln -s /usr/lib/spark-current/jars/scala-library-2.11.12.jar /usr/lib/hive-current/lib/scala-library-2.11.12.jar
    ln -s /usr/lib/spark-current/jars/spark-core_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-core_2.11-2.4.3.jar
    ln -s /usr/lib/spark-current/jars/spark-network-common_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-network-common_2.11-2.4.3.jar
    ln -s /usr/lib/spark-current/jars/spark-unsafe_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-unsafe_2.11-2.4.3.jar

    f. 测试一下是否正常

    2. HiveServer升级Hive包
    a. 登录emr-header-1, 将新的hive包解压到一个目录,比如/opt/apps/hive
    b. cp /usr/lib/hive-current/lib/emr-hive-hook*jar /opt/apps/hive/apache-hive-2.3.5-bin/lib
    c. unlink /usr/lib/hive-current
    d. ln -s /opt/apps/hive/apache-hive-2.3.5-bin /usr/lib/hive-current
    e. 建立其它软连接

    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-hive-plugin-impl /usr/lib/hive-current/lib/ranger-hive-plugin-impl
    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-hive-plugin-shim-1.2.0.jar /usr/lib/hive-current/lib/ranger-hive-plugin-shim-1.2.0.jar
    ln -s /opt/apps/ecm/service/ranger/1.2.0-1.1.0/package/ranger-1.2.0-1.1.0/ranger-1.2.0-1.1.0-hive-plugin/lib/ranger-plugin-classloader-1.2.0.jar /usr/lib/hive-current/lib/ranger-plugin-classloader-1.2.0.jar

    ln -s /usr/lib/spark-current/jars/scala-library-2.11.12.jar /usr/lib/hive-current/lib/scala-library-2.11.12.jar
    ln -s /usr/lib/spark-current/jars/spark-core_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-core_2.11-2.4.3.jar
    ln -s /usr/lib/spark-current/jars/spark-network-common_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-network-common_2.11-2.4.3.jar
    ln -s /usr/lib/spark-current/jars/spark-unsafe_2.11-2.4.3.jar /usr/lib/hive-current/lib/spark-unsafe_2.11-2.4.3.jar

    f. 登录emr-header-2,重复a~d的操作
    g. 页面控制台重启HiveServer
    h. 使用beeline/hue测试是否正常


    gateway上启动hs2: (可选)
    拷贝Ranger相关配置文件
    以root用户执⾏以下命令
    scp root@emr-header-1:/etc/ecm/hive-conf/ranger-hive-audit.xml /etc/ecm/hive-conf
    scp root@emr-header-1:/etc/ecm/hive-conf/ranger-hive-security.xml /etc/ecm/hive-conf
    scp root@emr-header-1:/etc/ecm/hive-conf/ranger-policymgr-ssl.xml /etc/ecm/hive-conf
    scp root@emr-header-1:/etc/ecm/hive-conf/ranger-security.xml /etc/ecm/hive-conf
    chown hadoop:hadoop /etc/ecm/hive-conf/ranger-*
    启动HiveServer2
    以hadoop用户执行以下命令

    HADOOP_OPTS=HIVE_SERVER2_OPTS HADOOP_HEAPSIZE=HIVE_SERVER2_HEAPSIZE hive --service
    hiveserver2 >/var/log/hive/hiveserver2.out 2>/var/log/hive/hiveserver2.err &

  • 相关阅读:
    面试基础知识文档
    敏捷式开发
    redis总结
    自我介绍的问题
    面试2
    唐巧的iOS技术博客选摘
    IOS开发中滑动页面时NSTimer停止的问题
    iOS多线程GCD(转)
    iOS 用instancetype代替id作返回类型有什么好处?
    C语言中全局变量、局部变量、静态全局变量、静态局部变量的区别 (转)
  • 原文地址:https://www.cnblogs.com/hongfeng2019/p/12120596.html
Copyright © 2020-2023  润新知