• 【原创】大数据基础之Ambari(3)通过Ambari部署Airflow


    ambari2.7.3(hdp3.1) 安装 airflow1.10

    ambari的hdp中原生不支持airflow安装,下面介绍如何通过mpack方式使ambari支持airflow安装:

    1 下载

    # wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz

    2 安装

    # ambari-server install-mpack --mpack airflow-service-mpack.tar.gz --verbose

    3 重启

    # ambari-server restart

    4 ambari页面没有出现ambari,检查mpack.json

    /var/lib/ambari-server/resources/mpacks/airflow-ambari-mpack-1.5.4/mpack.json

          "service_versions_map": [

            {

              "service_name" : "AIRFLOW",

              "service_version" : "1.10.0",

              "applicable_stacks" : [

                {

                  "stack_name" : "HDP",

                  "stack_version" : "2.4"

                },

                {

                  "stack_name" : "HDP",

                  "stack_version" : "2.5"

                },

                {

                  "stack_name" : "HDP",

                  "stack_version" : "2.6"

                },

                {

                  "stack_name" : "HDP",

                  "stack_version" : "3.0"

                }

              ]

            }

          ]

    需要增加HDP3.1,修改完重新打包

    4 卸载

    ambari-server uninstall-mpack --mpack-name=airflow-ambari-mpack

    5 安装(同上)

    6 重启(同上)

    打开ambari页面

    启动worker时可能失败,查看启动命令

    2019-01-18 19:09:48,159 - Execute['service airflow-worker start'] {}

    查看日志

    journalctl -xe

    发现有报错

    Jan 18 19:04:05 cdp-test-server-01.bj python[15246]: detected unhandled Python exception in '/var/lib/ambari-agent/cache/common-services/AIRFLOW/1.10.0/package/scripts/airflow_worker_control.py'
    Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: Executable '/var/lib/ambari-agent/cache/common-services/AIRFLOW/1.10.0/package/scripts/airflow_worker_control.py' doesn't belong to any package and ProcessUnpackaged is set t
    Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: 'post-create' on '/var/spool/abrt/Python-2019-01-18-19:04:05-15246' exited with 1
    Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: Deleting problem directory '/var/spool/abrt/Python-2019-01-18-19:04:05-15246'

    可以通过修改配置解决

    # grep ProcessUnpackaged /etc/abrt/abrt-action-save-package-data.conf
    no

    改为yes

    ambari具体使用详见:https://www.cnblogs.com/barneywill/p/10268501.html

    参考:

    https://github.com/miho120/ambari-airflow-mpack

  • 相关阅读:
    table标签中不能有div、span等标签
    奇葩的ASCII 160 号空格
    解决“只能通过Chrome网上应用商店安装该程序”的方法
    谈谈页面流程图(附案例) | 人人都是产品经理
    mysql-connector-java-5.1.34下载
    二、web.xml文件配置
    spring Beans初始化及配置
    Junit单元测试的使用
    什么是IOC?
    二、springMVC项目依赖配置pom.xml
  • 原文地址:https://www.cnblogs.com/barneywill/p/10284804.html
Copyright © 2020-2023  润新知