docker跨网段搭建gp测试环境

(1)制作docker镜像：

#docker run -i -d --name sdw1 -h sdw1 centos  #拉取一个基础镜像，然后在此基础上修改

　

容器里面操作：

#yum install vim wget compat-openssl10 langpacks-en glibc-all-langpacks readline-devel bzip2 krb5-devel perl rsync libevent apr apr-util openssl passwd libnsl net-tools which openssh-clients openssh-server less zip unzip iproute -y
#ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""
#ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""
#ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N "" #/usr/sbin/sshd

vi /etc/security/limits.conf
# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
vi /etc/sysctl.conf
#等一些简单设置以后,上传镜像到harbor仓库：　

然后拉取镜像开始测试：

申请2台ec2,ip为：

a机器  10-10-70-186     docker0 : 172.19.1.1

b机器 10-10-77-42       docker0 : 172.19.2.1
2台ec2的vpc网络放通，避免不必要的麻烦

(3)修改2台ec2默认docker0网卡ip地址(个人习惯)

修改/etc/docker/daemon.json文件，没有就编辑，修改后如下：

root@ip-10-10-70-186:~# cat /etc/docker/daemon.json
{"bip":"172.19.1.1/24"}
root@ip-10-10-77-42:~# cat /etc/docker/daemon.json
{"bip":"172.19.2.1/24"}

systemctl restart docker.service

(4)创建docker,写docker-compse.yaml配置文件：

root@ip-10-10-70-186:~# cat docker-compose.yml
version: '3'
services:
sdw3:
image: xxxx/ops/gpdb:v1.2
hostname: sdw3
container_name: db_sdw3
ports:
- "1225:22"
- "15435:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.3
sdw4:
image: xxxx/ops/gpdb:v1.2
hostname: sdw4
container_name: db_sdw4
ports:
- "1226:22"
- "15436:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.4
sdw5:
image: xxxx/ops/gpdb:v1.2
hostname: sdw5
container_name: db_sdw5
ports:
- "1227:22"
- "15437:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.3.5
networks:
mynet1:
ipam:
config:
- subnet: 172.19.3.0/24



root@ip-10-10-77-42:~# cat docker-compose.yml
version: '3'
services:
mdw:
image: xxxx/ops/gpdb:v1.2
hostname: mdw
container_name: db_mdw
ports:
- "1222:22"
- "15432:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.2
sdw1:
image: xxxx/ops/gpdb:v1.2
hostname: sdw1
container_name: db_sdw1
ports:
- "1223:22"
- "15433:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.3
sdw2:
image: xxxx/ops/gpdb:v1.2
hostname: sdw2
container_name: db_sdw2
ports:
- "1224:22"
- "15434:5432"
stdin_open: true
tty: true
networks:
mynet1:
ipv4_address: 172.19.4.4
networks:
mynet1:
ipam:
config:
- subnet: 172.19.4.0/24

2台ec2启动docker-compose正常启动docker：


然后配置互信，进入db_mdw容器：

root@10-10-77-42:~# docker exec -ti db_mdw /bin/bash
[root@mdw /]# su - gpadmin
Last login: Thu Apr  8 14:52:59 UTC 2021 on pts/2
[gpadmin@mdw ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/gpadmin/.ssh/id_rsa): 

...

配置mdw主机host文件：

[gpadmin@mdw ~]$ cat /etc/hosts
127.0.0.1       localhost

172.19.4.2 mdw
172.19.4.3 sdw1
172.19.4.4 sdw2
172.19.3.3 sdw3
172.19.3.4 sdw4
172.19.3.5 sdw5

然后再2台宿主机添加路由使2个宿主机内部docker能互访：



实现： 路由表方式：
a机器添加路由：route add -net 172.19.2.0/24 gw 10.10.77.42
b机器添加路由：route add -net 172.19.1.0/24 gw 10.10.70.186


a机器添加转发：iptables -t nat -A POSTROUTING -s 172.19.1.0/24 -d 172.19.2.0/16 -j MASQUERADE
iptables -t nat -I PREROUTING -s 172.19.3.0/24 -d 172.19.4.0/24 -j DNAT --to 172.19.1.1

b机器添加转发：iptables -t nat -A POSTROUTING -s 172.19.2.0/24 -d 172.19.1.0/16 -j MASQUERADE
iptables -t nat -I PREROUTING -s 172.19.4.0/24 -d 172.19.3.0/24 -j DNAT --to 172.19.2.1

操作以后，6个docker能互访，当时a机器内部docker与b机器的docker互信始终不行，折腾了许久没搞定，于是换了一种方法达到这6个docker互访没有问题。

使用https://github.com/weaveworks/weave这个工具充当路由器的功能。实现6个docker互访。

2台ec2分别安装weave工具：
curl -L git.io/weave -o /usr/local/bin/weave

root@10-10-70-186:~#

weave connect 10.10.77.42 to ## 连接另外一台ec2
weave attach 172.19.44.5/24 cd9754fd68b0 #db_sdw3
weave attach 172.19.44.6/24 ff2bddf34119 #db_sdw4
weave attach 172.19.44.6/24 ff2bddf34119 #db_sdw5

root@10-10-77-42:~#

weave connect to 10.10.70.186 ## 连接另外一台ec2
weave attach 172.19.44.2/24 ac3077ebfd0a #db_mdw
weave attach 172.19.44.3/24 b6349ed1305d #db_sdw1
weave attach 172.19.44.4/24 d51d1cbd7658 #db_sdw2

这样6个docker就实现了互访。

(5) 开始安装greenplum数据库:

root@10-10-77-42:~# docker exec -ti db_mdw /bin/bash
[root@mdw /]# su - gpadmin
##ssh-keygen设置ssh配置
##配置host文件
[gpadmin@mdw ~]$ cat /etc/hosts
127.0.0.1 localhost
172.19.44.2 mdw
172.19.44.3 sdw1
172.19.44.4 sdw2
172.19.44.5 sdw3
172.19.44.6 sdw4
172.19.44.7 sdw5

因为已经用weave改变了6个docker的ip，才能互访，所以用修改后的ip地址，之前docker-compse.yaml里面的ip可保留也可以丢弃。

[gpadmin@mdw ~]$ cat seg_hosts # 我暂时用4个docker搭建机器，留2个用来添加删除segment　

然后开始安装gp数据库

确保6个docker互信没有问题才可以安装数据库。

gp初始化参数如下：

[gpadmin@mdw ~]$ cat gpinitsystem_config
ARRAY_NAME="Greenplum"
MACHINE_LIST_FILE=/home/gpadmin/seg_hosts
# Segment 的名称前缀
SEG_PREFIX=gpseg
# Primary Segment 起始的端口号
PORT_BASE=33000
# 指定 Primary Segment 的数据目录
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatap1 /home/gpadmin/gpdata/gpdatap2)
# Master 所在机器的 Hostname
MASTER_HOSTNAME=mdw
# 指定 Master 的数据目录
MASTER_DIRECTORY=/home/gpadmin/gpdata/gpmaster
# Master 的端口
MASTER_PORT=5432
# 指定Bash的版本
TRUSTED_SHELL=/usr/bin/ssh
# Mirror Segment起始的端口号
MIRROR_PORT_BASE=43000
# Primary Segment 主备同步的起始端口号
REPLICATION_PORT_BASE=34000
# Mirror Segment 主备同步的起始端口号
MIRROR_REPLICATION_PORT_BASE=44000
# Mirror Segment 的数据目录
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatam1 /home/gpadmin/gpdata/gpdatam2)

[gpadmin@mdw ~]$ gpssh -f seg_hosts

=>

mkdir -p /home/gpadmin/gpdata/gpmaster/gpseg-1
mkdir -p /home/gpadmin/gpdata/gpdatap1
mkdir -p /home/gpadmin/gpdata/gpdatam1
mkdir -p /home/gpadmin/gpdata/gpdatam2
mkdir -p /home/gpadmin/gpdata/gpdatap2

chmod 777 -R /home/gpadmin/gpdata
chown gpadmin.gpadmin -R /home/gpadmin/gpdata

echo 'export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/gpmaster/gpseg-1' >> /home/gpadmin/.bashrc

这样在4个docker均已创建好相应目录已经设置好权限。

for i in $(docker ps -a -q);do docker exec -ti $i ip a|grep inet|awk '{print $2}'|grep -vE '^fe80|^::|^127'|awk -F '/' '{print $1}';done  # 2台ec2宿主机执行，检查docker ip地址

[gpadmin@mdw ~]$ gpinitsystem -c gpinitsystem_config

。。。。

最后完成。



最大的坑，安装的时候报错，排查安装日志报错提示：

/usr/local/greenplum-db-6.15.0/bin/psql: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory


解决方法：
ls -la /usr/lib64 |grep libreadline
[gpadmin@660f628171a7 gpAdminLogs]$ ldd /usr/local/greenplum-db-6.15.0/bin/psql
linux-vdso.so.1 (0x00007ffde69ca000)
libpq.so.5 => /usr/local/greenplum-db-6.15.0/lib/libpq.so.5 (0x00007fab0fbb4000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fab0f945000)
libreadline.so.6 => not found
libc.so.6 => /lib64/libc.so.6 (0x00007fab0f582000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fab0f11f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fab0eeca000)
libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007fab0ec73000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fab0ea53000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fab0e84f000)
libz.so.1 => /lib64/libz.so.1 (0x00007fab0e638000)
/lib64/ld-linux-x86-64.so.2 (0x00007fab0fdea000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fab0e34f000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fab0e138000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fab0df34000)

ln -s /usr/lib64/libreadline.so.7.0 /usr/lib64/libreadline.so.6