• 大数据平台搭建（Ambari +HDP）

    安装包

    • ambari-2.7.3.0-centos7.tar.gz
    • HDP-3.1.0.0-centos7-rpm.tar.gz
    • HDP-UTILS-1.1.0.22-centos7.tar.gz
    • HDP-GPL-3.1.0.0-centos7-gpl.tar.gz

    Ambari

    • Ambari 跟 Hadoop 等开源软件一样，也是 Apache Software Foundation 中的一个项目，并且是顶级项目。就 Ambari 的作用来说，就是创建、管理、监视 Hadoop 的集群，但是这里的 Hadoop 指的是 Hadoop 整个生态圈（例如 Hive，Hbase，Sqoop，Zookeeper 等）， 而并不仅是特指 Hadoop。用一句话来说，Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个管理工具。

    • Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。简单来说，用户通过 Ambari Server 通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI，方便用户了解到集群的各种状态，并进行相应的维护。

    HDP

    • HDP 是 hortonworks 的软件栈，里面包含了 Hadoop 生态系统的所有软件项目，比如 HBase、Zookeeper、Hive、Pig 等

    HDP-UTILS

    • HDP-UTILS 是工具类库

    准备工作

    集群节点规划

    主机名 IP 功能 内存 磁盘
    yum.hdp 192.168.229.30 Ambari/HDP packages 1G 50G
    node-01.hdp 192.168.229.31 Ambari Server/Agent 3G 50G
    node-02.hdp 192.168.229.32 Ambari Agent 2G 50G
    node-03.hdp 192.168.229.33 Ambari Agent 2G 50G

    克隆 4 台虚拟机

    image-20210524232113413

    修改主机名
    # vi /etc/hostname
    修改 IP 地址
    # vi /etc/sysconfig/network-scripts/ifcfg-ens33
# service network restart
# ping www.baidu.com
    hosts 文件添加主机名和IP地址映射
    • windows 系统

    hosts 文件路径：C:WindowsSystem32driversetc

    192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 ndoe-03.hpd
    • centos 系统
    [root@node-01 ~]# vi /etc/hosts
192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 node-03.hdp
    配置免密
    [root@yum ~]# ssh-keygen -t rsa
[root@yum ~]# ssh-copy-id yum.hdp
[root@yum ~]# ssh-copy-id node-01.hdp
[root@yum ~]# ssh-copy-id node-02.hdp
[root@yum ~]# ssh-copy-id node-03.hdp

[root@node-01 ~]# ssh-keygen -t rsa
[root@node-01 ~]# ssh-copy-id node-01.hdp
[root@node-01 ~]# ssh-copy-id node-02.hdp
[root@node-01 ~]# ssh-copy-id node-03.hdp

[root@node-02 ~]# ssh-keygen -t rsa
[root@node-02 ~]# ssh-copy-id node-02.hdp
[root@node-02 ~]# ssh-copy-id node-03.hdp
    安装 mysql

    在 node-01.hdp 主机节点上安装 mysql，安装教程

    在 yum.hdp 上安装 Ambari

    安装 yum 相关工具

    [root@yum ~]# yum install yum-utils -y
[root@yum ~]# yum repolist
[root@yum ~]# yum install createrepo -y

    上传安装包到 yum.hdp

    创建 hdp 目录，将安装包上传到该目录下

    [root@yum ~]# mkdir hdp

    image-20210622213025354

    安装 Apache httpd

    httpd 为 Apache HTTP 服务器程序，直接执行程序可启动服务器的服务

    [root@yum ~]# yum install httpd -y

    • 安装完成后，会生成 /var/www/html 目录（相当于 Tomcat 的 webapps 目录），进入到 /var/www/html 目录下，创建 ambari 和 hdp 目录，用来存放安装文件

      [root@yum ~]# cd /var/www/html/
[root@yum html]# mkdir ambari hdp hdp-utils hdp-gpl

[root@yum ~]# cd /root/hdp/
[root@yum hdp]# tar -zxvf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/ambari/
[root@yum hdp]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp-utils
[root@yum hdp]# tar -zxvf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
[root@yum hdp]# tar -zxvf HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/hdp-gpl/

    • 启动 httpd 服务

      # 启动 httpd 服务
[root@yum ~]# systemctl start httpd
 Active: active (running) since 三 2021-05-26 15:24:34 CST; 16s ago #表示服务正在运行
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 1247 (httpd)
# 查看 httpd 服务状态
[root@yum ~]# systemctl status httpd
# 设置 httpd 服务开机启动
[root@yum ~]# systemctl enable httpd

    • 默认端口80，浏览器输入：http://192.168.229.30

    image-20210526152824074

    image-20210526152943610

    image-20210526153017273

    配置本地 Repo 数据源

    • 创建 ambari.repo 配置文件

      [root@yum yum]# cd /etc/yum.repos.d
[root@yum yum.repos.d]# vim ambari.repo

#VERSION_NUMBER=2.7.3.0-139
[ambari-2.7.3.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.3.0
#baseurl=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0
baseurl=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

    • 创建 hdp.repo 配置文件

      [root@yum yum]# cd /etc/yum.repos.d
[root@yum yum.repos.d]# vim hdp.repo

#VERSION_NUMBER=3.1.0.0-78
[HDP-3.1.0.0]
name=HDP Version - HDP-3.1.0.0
#baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0
baseurl=http://yum.hdp/hdp/HDP/centos7
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
#baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7
baseurl=http://yum.hdp/hdp-utils/HDP-UTILS/centos7
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/hdp-utils/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1

[HDP-GPL-3.1.0.0]
name=HDP-GPL Version - HDP-GPL-3.1.0.0
baseurl=http://yum.hdp/hdp-gpl/HDP-GPL/centos7
gpgcheck=1
gpgkey=http://yum.hdp/hdp-gpl/HDP-GPL/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

    • 分发 ambari.repo 和 hdp.repo 配置文件

      [root@yum yum]# cd /etc/yum.repos.d
[root@yum yum]# scp ambari.repo node-01.hdp:$PWD
[root@yum yum]# scp ambari.repo node-02.hdp:$PWD
[root@yum yum]# scp ambari.repo node-03.hdp:$PWD
[root@yum yum]# scp hdp.repo node-01.hdp:$PWD
[root@yum yum]# scp hdp.repo node-02.hdp:$PWD
[root@yum yum]# scp hdp.repo node-03.hdp:$PWD

    • 生成本地源

      使用 createrepo 命令，创建 yum 本地源（软件仓库），即为存放本地特定位置的众多rpm包建立索引，描述各包所需依赖信息，并形成元数据

      [root@yum yum]# createrepo /var/www/html/hdp/HDP/centos7/
Spawning worker 0 with 101 pkgs
Spawning worker 1 with 100 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

[root@yum yum]# createrepo /var/www/html/hdp-utils/HDP-UTILS/centos7/
Spawning worker 0 with 8 pkgs
Spawning worker 1 with 8 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

[root@yum yum.repos.d]# createrepo /var/www/html/hdp-gpl/HDP-GPL/centos7/
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

    在 node-01.hdp 节点上 mysql 上创建相应的用户和数据库

    • 创建 ambari 数据库及数据库的用户名和密码

      [root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'ambari'@'%'IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected (0.00 sec)

mysql>  GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

    • 创建hive数据库及hive库的用户名和密码

      [root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database hive character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'hive'@'%'IDENTIFIED BY 'Hive123';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

    • 创建oozie数据库及oozie库的用户名和密码

      [root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database oozie character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'oozie'@'%'IDENTIFIED BY 'Oozie123';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

    • 创建ranger数据库及ranger库的用户名和密码（不用创建）

      [root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database ranger character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'rangeradmin'@'%'IDENTIFIED BY 'Ranger123';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ranger.* TO 'rangeradmin'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

    • 查看所有创建的数据库

      mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ambari             |
| hive               |
| mysql              |
| oozie              |
| performance_schema |
| ranger             |
| sys                |
+--------------------+
8 rows in set (0.00 sec)

    • 下载 mysql-connection-java

      [root@node-01 mysql]# yum install mysql-connector-java

#查看是否下载 msyql 驱动 jar包
[root@node-01 mysql]# ls /usr/share/java | grep mysql
mysql-connector-java.jar

    在 node-01.hdp 节点上安装 Ambari-Server

    [root@node-01 ~]# yum install ambari-server
[root@node-01 mysql]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'enforcing'
Temporarily disabling SELinux
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /root/apps/jdk1.8.0_141
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-01.hdp
Port (3306): 3306
Database name (ambari): ambari
Username (root): ambari
Enter Database Password (root):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
    • 使用 ambari 登录数据库
    [root@node-01 mysql]# mysql -uambari -p
Enter password: #密码为Ambari123
mysql> use ambari;
Database changed
#执行 sql 脚本创建ambari数据库表
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
#查看数据库表是否创建成功
mysql> show tables;

    启动Ambari-Server 服务

    [root@node-01 mysql]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start........................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

    停止 Ambari-Server 服务： # ambari-server stop

    安装Ambari-Agent

    [root@node-01 ~]# yum -y install ambari-agent
[root@node-02 ~]# yum -y install ambari-agent
[root@node-03 ~]# yum -y install ambari-agent

    访问Ambari web页面

    默认端口8080，Username：admin；Password：admin；http://192.168.229.31:8080

    image-20210622141320754

    开始安装集群

    • 启用安装向导创建集群

    image-20210622141529121

    • 配置集群名称

    image-20210622141811342

    • 选择版本

    image-20210622215235614

    YUM 本地源如下：

    HDP-3.1：http://yum.hdp/hdp/HDP/centos7

    HDP-3.1-GPL:http://yum.hdp/hdp-gpl/HDP-GPL/centos7

    HDP-UTILS-1.1.0.22:http://yum.hdp/hdp-utils/HDP-UTILS/centos7

    • 配置集群节点、密钥

    image-20210622145039033

    • 主机确认

    image-20210622161750539

    • 选择大数据组件

    image-20210622163644290

    • 节点分配

    image-20210622164141232

    • 分配从属与客户端

    image-20210622164241261

    • 定制服务

    image-20210622164514401

    image-20210622165533252

    #重新执行命令
[root@node-01 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.

    image-20210622165714329

    image-20210622170000323

    image-20210622170035967

    • 集群整体部署

    image-20210622170158555

    image-20210622170216482

    • 开始安装（漫长的等待...）

    image-20210622222925308

    • 安装完毕

      不用担心警告，后期可以调整，搭建完成，可以在展示页面进行查看集群状态

    image-20210622171135378

    image-20210622170436501

    • 查看监控界面

    image-20210622170519948

    • 查看服务状态
    作者：Binge
    本文版权归作者和博客园共有，转载必须给出原文链接，并保留此段声明，否则保留追究法律责任的权利。
  • 相关阅读:
    吊打996，来了？！
    微软开源浏览器自动化工具Playwright for Python（附源码）
    从0到1开始建设安全测试体系
    网友爆料vivo将取消大小周，不降薪，官方证实消息属实
    认识了一个在华为任职的50岁程序员！
    漫画：什么是自动驾驶？
    中国十大杰出人物
    最难调试修复的 bug 是怎样的？
    “十年不升职多得是，四年算什么”
    appium+pytest实现APP并发测试
  • 原文地址：https://www.cnblogs.com/binbingg/p/14921225.html
Copyright © 2020-2023  润新知