HDP集群部署

一、环境准备

1. centos7 安装和系统环境处理
   1）运行初始化脚本

!/bin/bash

-------------------------------------

系统环境初始化脚本

本脚本仅支持centos， centos7.7验证成功

使用root用户运行

------------------------------------

1 依赖安装

yum clean all
yum makecache
yum update -y
yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd svn ncurses-devel bzip2 psmisc libxslt-devel libxml2-devel libxslt.so.1 cyrus-sasl-plain yum groupinstall chinese-support mod_ssl ntp

2 关闭防火墙

echo "close firewall ----------------------------------------------------------"
firewall-cmd --state
systemctl status firewalld.service
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service

3 关闭SELINUX

echo "close SELINUX ----------------------------------------------------------"
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' '/etc/selinux/config'

4 修改UMASK

echo "set umask 0022 ----------------------------------------------------------"
umask 0022
echo umask 0022 >> /etc/profile

5 关闭swappiness

swap_dev=swapon -s | awk '{if (NR>1){print $1}}'
echo "swap device is ${swap_dev} ----------------------------------------------------------"
echo "close swap device ${swap_dev} ----------------------------------------------------------"
swapoff ${swap_dev}
if [ "$?" != 0 ] ; then
echo "close swap device ${swap_dev} failed ----------------------------------------------------------"
else
echo "close swap device ${swap_dev} success ----------------------------------------------------------"
fi

6 资源限制

echo "set ulimit 102400 ----------------------------------------------------------"
ulimit -Hn 102400
ulimit -Sn 102400
echo "* - nproc 102400" >> /etc/security/limits.conf
echo "* - nofile 102400" >> /etc/security/limits.conf

7 时钟同步

echo "start ntp ----------------------------------------------------------"
systemctl start ntpd
systemctl enable ntpd
systemctl status ntpd

8 禁止huge_page

echo "close huge_page ----------------------------------------------------------"
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

9 生成公钥私钥

echo "ssh-keygen ----------------------------------------------------------"
mkdir ~/.ssh
ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
2）如果时区不合适，需要修改时区
[root@centos ~]# timedatectl
3）hostname 修改
[root@centos ~]# hostnamectl set-hostname hdp-master01
4）添加hosts
vim /etc/hosts
5）免密登陆
ambari-server root用户ssh免密登陆所有其他节点
[root@centos ~]# ssh-copy-id root@node03
2. java8
在 /etc/profile.d/目录下，增加java.sh

!/bin/bash

java.sh

JAVA_HOME=/usr/local/services/jdk1.8.0_191
JRE_HOME=/usr/local/services/jdk1.8.0_191/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
3. python2.7.5
yum -y install python-pip
pip install --upgrade pip
4. mysql
Node03:3306
root/12345678
yum install mysql-connector-java*
5. 目录规划
mkdir /hadoop
将所有磁盘挂载或者链接到此目录下
/hadoop/disk01
/hadoop/disk02
二、建设本地yum repo
有网络可以直接通过网络，无网络需要先拷贝所有需要下载的文件

依赖安装

yum install yum-utils createrepo httpd

创建目录

mkdir -p /var/www/html
mkdir -p /var/www/html/ambari
mkdir -p /var/www/html/hdp

下载repo url文件

cd /etc/yum.repos.d/
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/hdp.repo

查看repo

yum repolist

下载repo文件到本地

cd /var/www/html/ambari
reposync -r ambari-2.7.4.0
cd /var/www/html/hdp
reposync -r HDP-3.1.4.0
reposync -r HDP-UTILS-1.1.0.22

新建本地repo

createrepo /var/www/html/ambari/ambari-2.7.4.0
createrepo /var/www/html/hdp/HDP-3.1.4.0
createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.22

启动http服务

systemctl start httpd.service
systemctl enable httpd.service
修改repo web地址
[root@node03 ~]#vim /etc/yum.repos.d/ambari.repo
[ambari-2.7.4.0]
baseurl=http://node03/ambari/ambari-2.7.4.0
...
[root@node03 ~]#vim /etc/yum.repos.d/hdp.repo
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://node03/hdp/HDP-3.1.4.0
...
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://node03/hdp/HDP-UTILS-1.1.0.22
...
/etc/yum.repos.d/ambari.repo和hdp.repo需要拷贝到ambari-server将要部署的服务器
三、部署ambari-server
1.准备数据库
CREATE DATABASE ambari default character set utf8mb4 collate utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON . TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
1.安装ambari-server
yum install ambari-server
2.设置
[root@node01 /etc/yum.repos.d]#ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root

Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK

Enter choice (1): 2
Path to JAVA_HOME: /usr/local/services/jdk1.8.0_191

GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Enter advanced database configuration [y/n] (n)? y
Configuring database...

Choose one of the following options:
[3] - MySQL / MariaDB

Enter choice (1): 3
Hostname (localhost): node03
Port (3306): 3306
Database name (ambari): ambari
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password: 12345678

Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
3.建表
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
4.启动

端口 8080

ambari-server start

ambari-server status
ambari-server stop
ambari-server restart
四、添加组件
注册 admin/admin
创建集群

选择版本设置仓库

添加节点

上传ambari-server所在服务器的私钥id_rsa
配置用户和ssh端口
使用的host不是域名，下一步
注册所有节点

添加组件

选择组件实际部署节点

设置grafana 帐号密码以及Activity 密码

设置文件目录（需要提前规划）

设置各个组件的启动用户

HDFS服务设置

部署

五、测试

测试hdfs 上传文件

hadoop fs mkdir /tmp/test/in
hadoop fs -put text.txt /tmp/test/in

测试yarn 运行任务

hadoop jar /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /tmp/test/in /tmp/test/out