我们先克隆几台机器
我们打开克隆出来的机器
我们先把主机名修改一下
我们把主机名改成am2
下一步我们来配置网卡
把原来的eth0的注释掉，把现在的eth1改成eth0,同时把mac地址记下来
保存退出
把ip修改一下
然后重启机器，我们可以看到主机名改了，网络也通了
同样的操作我们再克隆两台机器，这次搭建集群我们一共4台机器，具体我就不再多说了
下面我们来关闭防火墙
我们的防火墙是开启的
我们现在把防火墙关闭
现在只是临时关闭
下面我们要永久关闭
下面禁用SELINUX
vim /etc/sysconfig/selinux
关闭防火墙的工作完成，也是4台机器都要这样操作
下面我们按安装一下相关工具（4台机器都安装）
yum install lrzsz
yum install -y openssh-clients
下面我们配置时钟同步
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
yum install -y ntp 给每台机器都安装
下面我们用这个命令ntpdate pool.ntp.org给每台机器做一下时钟同步
修改hosts文件（4台机器都做）
vim /etc/hosts
对于自己的电脑windows下
配置免密码登录
在4台机器都输入ssh-keygen -t rsa
4个节点都这样操作，也节点1为例
[root@am1 ~]# cd .ssh [root@am1 .ssh]# ls id_rsa id_rsa.pub [root@am1 .ssh]# cat id_rsa.pub >> authorized_keys [root@am1 .ssh]#
下面对文件赋予权限（4个节点都做）
[root@am1 .ssh]# cd .. [root@am1 ~]# chmod 700 .ssh [root@am1 ~]# chmod 600 .ssh/* [root@am1 ~]#
把所有的公钥到拷贝到am1节点去
cat ~/.ssh/id_rsa.pub | ssh root@am1 'cat >> ~/.ssh/authorized_keys'
现在把am1的authorized_keys文件分发到其他三个节点
[root@am1 .ssh]# scp -r authorized_keys root@am2.djt.com:~/.ssh/ ssh: Could not resolve hostname am2.djt.com: Name or service not known lost connection [root@am1 .ssh]# scp -r authorized_keys root@am2:~/.ssh/ The authenticity of host 'am2 (192.168.86.42)' can't be established. RSA key fingerprint is 67:b6:14:54:d2:1d:f7:45:d6:30:3e:86:39:1d:82:65. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'am2,192.168.86.42' (RSA) to the list of known hosts. root@am2's password: authorized_keys 100% 1950 1.9KB/s 00:00 [root@am1 .ssh]# scp -r authorized_keys root@am3:~/.ssh/ The authenticity of host 'am3 (192.168.86.43)' can't be established. RSA key fingerprint is 67:b6:14:54:d2:1d:f7:45:d6:30:3e:86:39:1d:82:65. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'am3,192.168.86.43' (RSA) to the list of known hosts. root@am3's password: authorized_keys 100% 1950 1.9KB/s 00:00 [root@am1 .ssh]# scp -r authorized_keys root@am4:~/.ssh/ The authenticity of host 'am4 (192.168.86.44)' can't be established. RSA key fingerprint is 67:b6:14:54:d2:1d:f7:45:d6:30:3e:86:39:1d:82:65. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'am4,192.168.86.44' (RSA) to the list of known hosts. root@am4's password: authorized_keys
下面我们按照jdk1.8
把文件分发给其他3个节点
scp -r jdk1.8.0_60/ root@am2:/home/app/
scp -r jdk1.8.0_60/ root@am3:/home/app/
scp -r jdk1.8.0_60/ root@am4:/home/app/
配置环境变量 vim /etc/profile
#java export JAVA_HOME=/home/app/jdk1.8.0_60 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin
使环境变量生效
source /etc/profile
另外的三个节点也一样的，这里不多赘述了
现在开始部署本地库
https://zh.hortonworks.com/
https://docs.hortonworks.com/index.html
https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_Installing_HDP_AMB/content/_ambari_repositories.html
下载
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.1.0/ambari-2.2.1.0-centos6.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.0.0/HDP-2.4.0.0-centos6-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz
用这个命令sudo yum install yum-utils createrepo来创建本地的yum 源
安装Apache
sudo yum install httpd
通过这个工具把刚刚下载的三个大文件上传到这个目录下
再创建目录
解压
把这几个包复制到我们后面建的目录里面
[root@am1 2.2.1.0-161]# cp -r /var/www/html/AMBARI-2.2.1.0/centos6/2.2.1.0-161/ambari/ /var/www/html/hdp2.4/centos6/ [root@am1 2.2.1.0-161]# cp -r /var/www/html/AMBARI-2.2.1.0/centos6/2.2.1.0-161/RPM-GPG-KEY/ /var/www/html/hdp2.4/centos6/ [root@am1 2.2.1.0-161]# cp -r /var/www/html/AMBARI-2.2.1.0/centos6/2.2.1.0-161/smartsense/ /var/www/html/hdp2.4/centos6/ [root@am1 2.2.1.0-161]#
解压HDP-2.4.0.0-centos6-rpm.tar.gz
我们先把已经解压的包删除掉来节省空间
我们进HDP目录来看看
我们一样的把这些蓝色红色的包都放到目录 /var/www/html/hdp2.4/centos6/ 下
[root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/accumulo/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/atlas/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/bigtop-jsvc/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/bigtop-tomcat/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/calcite/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/datafu/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/falcon/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/flume/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hadoop/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hadooplzo/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hbase/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/HDP-2.4.0.0-centos6-m2-artifacts.tar /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hdp-select/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hive/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/hue/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/kafka/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/knox/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/mahout/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/oozie/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/phoenix/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/pig/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/ranger/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/RPM-GPG-KEY/ /var/www/html/hdp2.4/centos6/ cp: overwrite `/var/www/html/hdp2.4/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins'? y [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/slider/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/slider-app-packages/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/spark/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/sqoop/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/storm/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/tez/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/vrpms/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/zeppelin/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]# cp -r /var/www/html/HDP/centos6/2.x/updates/2.4.0.0/zookeeper/ /var/www/html/hdp2.4/centos6/ [root@am1 2.4.0.0]#
解压 HDP-UTILS-1.1.0.20-centos6.tar.gz
我们把这些都拷贝到目录 /var/www/html/hdp2.4/centos6/ 下
[root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/epel-release-6-8.noarch.rpm /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/extjs/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/fping/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/ganglia/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/hadoop-lzo/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/libconfuse/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/lucid/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/lzo/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/mysql-connector-java/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/nagios/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/nagios-plugins/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/pdsh/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/perl/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/RPM-GPG-KEY/ /var/www/html/hdp2.4/centos6 cp: overwrite `/var/www/html/hdp2.4/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins'? y [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/rrdtool/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]# cp -r /var/www/html/HDP-UTILS-1.1.0.20/repos/centos6/snappy/ /var/www/html/hdp2.4/centos6 [root@am1 centos6]#
复制完了就这个结果
执行sudo createrepo /var/www/html/hdp2.4/centos6命令生成repodata
启动HTTP服务
在浏览器输入http://192.168.86.41/hdp2.4/地址
可以点进来看看
下面我们修改一下我们本地源的地址
我们在这里安装本地源
先下载wget
sudo wget -c http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.1.0/ambari.repo
来下载源
下面我们来修改一下这个文件
#VERSION_NUMBER=2.2.1.0-161 [Updates-ambari-2.2.1.0] name=ambari-2.2.1.0 - Updates baseurl=http://192.168.86.41/hdp2.4/centos6/ gpgcheck=0 gpgkey=http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
安装并配置ambari-sever
检查仓库是否可用
证明是可用的
安装我们的ambari-server
接下来我们来配置我们的ambari-server
sudo ambari-server setup
[root@am1 yum.repos.d]# sudo ambari-server setup Using python /usr/bin/python2 Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):root Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /home/app/jdk1.8.0_60 Validating JDK on Ambari Server...done. Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? n Configuring database... Default properties detected. Using built-in database. Configuring ambari database... Checking PostgreSQL... Running initdb: This may take upto a minute. Initializing database: [ OK ] About to start PostgreSQL Configuring local database... Connecting to local database...connection timed out...retrying (1) Connecting to local database...done. Configuring PostgreSQL... Restarting PostgreSQL Extracting system views... .ambari-admin-2.2.1.0.161.jar ..... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. [root@am1 yum.repos.d]#
启动ambari-server
[root@am1 yum.repos.d]# sudo service ambari-server status Using python /usr/bin/python2 Ambari-server status Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid [root@am1 yum.repos.d]# sudo service ambari-server start Using python /usr/bin/python2 Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Ambari Server 'start' completed successfully. [root@am1 yum.repos.d]#
输入地址http://192.168.86.41:8080/
输入用户名和密码，都是admin
下面我们来搭建我们的集群
-----BEGIN RSA PRIVATE KEY----- MIIEoQIBAAKCAQEAsAAETlLM0iGpUQmt69toNuIy2sAPSPB5JrPwY8aj7OAaFHI+ kCmFCeppOcFFkH8ZP3d34OONP/Flc3R/dVqcablKW9XdYJNGyRHYFuQHxQt5ryhk mowoXBGGHY4yk1lBpy2F/QK57fxe/9n5HMcAocHllFiQ+9Drpl11NYwUG7CaroJb BNCPH14HdMF4InDvaKwB+GfRgQniWyEEBjGUC45xA55+dmx58P34/L53msmKtpES 350BUt+rktZzAgSjFD6g8A2t+aVB5NBvWPlgSnXGPXbpb/7k4rvim6sxZaJPG6Ht apV5ClqKqgGO1BIkiVBcn0qnOLnpvgLRNCQ2MQIBIwKCAQBag6pxbGlWH/CtVW9j WuUjikYKGaF1+APKwvCoV4eBIsuVfJU0Mpww/DYds9qiFXqy7PvSvititqk0EAcJ J0kgbeu6M3kqWl7rEH3CoSiRORoCT0mukTlT68FRB0011iHDsQMUZ81V0j98RDb4 219pIePIopO0tJZy0PpzT1rMaJm3EefM3M987f7J4Ck9OxhQAgYa7auo6YXqNmfF ETCLTwx+U/n591AgGG4VwcGnRgq9ZaWkOZm6QlURJ97C0fIfDqt6sKQqfTse0mIi gIkMTvRi3W0O2ANiudHXMxLIRaAbGKmy2P/NJHN6zOpNECGgZyP1JIkktnHT0+tX urVjAoGBANoFVZsfhvqBKvMe+MaZ8VRnly9r1ZuG+FKcQHCxZEYfjU8TZZtblba/ crQh63f+/IKrm2WpoD9yYj+hl0IVeBengvAh0mayk3bqdEgjnkl8I6YV7/2c0tF4 KKq4dARwEaPG2FrpWscgdLrqxjaOnbirw7AU4xRLCekAS4A803Q3AoGBAM6owtKB nDoO1hawzbat4ggsnxwC4ISjZ+7pPeYASFMWA976lZl756dk6f0F2j/y/bpFWrI9 xaUvNVTBqo3/9FET5kbt488tVy1a34TV0u/hDe+gLbQ2TylHDXUyjMFTs/QDp9bB 6hXbAn1s5cmTKmNeizv02WvQ9+qFYlywSUTXAoGBAI9FVYMx+Z1NkUCt8/A5RtEQ 37Fyv5IW3bKhMakkHVKnBRa0+aCbRSBR7Ep8sK3wtJBiJEofAufdc8NqMDK9prB1 ZKxt/0rNHxOhYltKmzeauFcy/Ms7K3O1XI1x5dcH0RPacOtI4+Hwwbyo6KeCSmNp jzkyTBSmVvgzZNCc//vpAoGAcC+6N8Kz5QC9XMZhC2W1N6MxzWClxFi0w4XuZuo1 4/YCGfW+7Ot9wUVpFFOiXTrEQIwMqecE5KSZS0SPyW2wg8j5WbRXGLI2o5e7Ktp5 0q1fVjJh9B17bi3iu/beshd+8i3exQouniZvEONYK5kIYdQ89Kl9V8k2IDnHrqjX UUECgYAVUFsd5Hkg1iIlN5UD8Lo6pVwKLdEDKFt0wmkdtOvbM+Uot3lI2oCk45MN YKat1yyEdl+kT6JA7zK7he/bVAQLnEZQ68q89i4jNYzoWTOwX5WEOCgD6uwY7Q6N T9EfzxYXAZ2R4M0ktS0rvXYIcjJVsLa19G6/8oBvi711dPieOA== -----END RSA PRIVATE KEY-----
我们可以看到报错了，
我们检查一下openssl的版本
记得刷新页面
这个是因为openssl的版本太低的原因，我们随便把其他节点也更新了
sudo yum install openssl
我这里做一下扩展，如果有的朋友用了centos7版本的话也会遇到这样的问题，可能重新更新openssl之后一样报出同样的错误，让你检查openssl版本，
如果是这样的话就这样解决
把这个值改成disable
[root@slave1 ~]# vim /etc/python/cert-verification.cfg
[root@slave1 ~]#
# Possible values are: # 'enable' to ensure HTTPS certificate verification is enabled by default # 'disable' to ensure HTTPS certificate verification is disabled by default # 'platform_default' to delegate the decision to the redistributor providing this particular Python version # For more info refer to https://www.python.org/dev/peps/pep-0493/ [https] verify=disable ~
我们点开这里
可以看到有一些相关的警告
这个是时钟同步的问题
这样来解决
可以看到这个问题解决了
前的问题我们这样来解决
[root@am2 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag [root@am2 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled [root@am2 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@am2 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag [root@am2 ~]#
我们选择这三个
这几条命令我们在其他节点也执行一下
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
我们修改一下权限问题
同时我们也把另外几个节点的目录权限问题都修改一下
我们可以添加服务
现在我们单节点的跑成功了，下面我们来怎么添加机器
是因为ntpd服务没有启动
接下来我们配置HA
添加zookeeper
添加完了重启，
am4也是一样的
分别执行这两条命令
在am2节点执行这命令
分别运行，具体我就不多说了
