Linux ->> 搭设GreemPlum集群环境

最近开始学习MPP数据库技术，打算接触下GreenPlum。这里参照《Greenplum企业应用实战》这本书的环境搭设步骤，这里需要4台机器，一台作为master，3台slave。

实验环境软件安装准备：

VMWare 12虚拟机

Xshell 7远程服务器连接管理器

VMWare 网络设置：

这里选NAT 或者桥接都行，但是有个建议？如果你用的是公司的网络，不是自家私人电脑上实验，建议用NAT。如果是自家私人电脑实验，那就无所谓，可以建议桥接，桥接的特点是每台虚拟机都会模拟成跟主机一样的实体机器跟DHCP服务器申请IP，好处是可以访问实际机器所在网段所有的网络资源，但是如果是公司网络就不好管理你这几台机器，毕竟它们都只是虚拟机，实验用途。NAT是把你本机当初一台路由器，本机充当网关，分配一个网段给虚拟机用，好处是不影响物理网络的IP分配（如果你是公司网络，你好几台虚拟机，每台都要用固定地址，容易跟其他DHCP获取IP地址的物理机器冲突）。

打开VMWARE的虚拟网络编辑器，设置一下NAT模式的子网地址，再改下网关IP（默认网关IP都是X.X.X.2，我比较习惯254，改成254）

网段

实验环境机器:

机器　　　　机器名称　　IP地址　　　　　　操作系统
master　　　gpmaster　 192.168.223.150　　CentOS 7

slave1　　　gpslave1　　192.168.223.151　　CentOS 7
slave2　　　gpslave2　　192.168.223.152　　CentOS 7
slave3　　　gpslave3　　192.168.223.153　　CentOS 7

数据库存储：GreenPlum的主要用途是数据仓库或者OLAP系统数据库，数据仓库是非常吃IO的，基本上一条SQL都是全表扫描（所以列存储形式的表结构在OLAP系统数据库很常见），每次处理的数据量非常大，数据基本停留在内存或者缓冲区里面很短时间就会被冲刷出去（数据字典除外），全表扫描主要是顺序IO。一般数据库服务器的硬件资源瓶颈多数情况下来自于磁盘IO，所以一般服务器都会对磁盘做RAID，而RAID1+0是目前主流的数据库服务器的磁盘阵列选择，RAID1+0兼顾了读写速度和容灾，虽然容灾的能力不及RAID5，但是数据仓库一般很少用RAID5，比较耗费磁盘，因为RAID5起码要5块磁盘，比较浪费。这里我对4台机器分别添加4块磁盘做RAID1+0，文件系统选择XFS。这里主要讲GreenPlum搭建，具体可以去另外一篇文章看（Linux ->> 添加RAID1+0磁盘阵列）。

网络配置：网络配置方面主要要保证集群机器间能相互通信，由于是内网机器，这里只需要修改/etc/hosts文件，把几台内网机器的电脑名称添加到每天机器的/etc/hosts文件，确保防火墙关闭（避免IPTable把端口屏蔽了）。确保能相互ping通。

用户组和用户：创建用户组和用户，授权用户目录权限，修改密码

[root@gpmaster md10]# groupadd -g 530 gpadmin
[root@gpmaster md10]# useradd -g 530 -u 530 -m -d /home/gpadmin -s /bin/bash gpadmin
[root@gpmaster md10]# chown -R gpadmin:gpadmin /home/gpadmin/
[root@gpmaster md10]# passwd gpadmin
更改用户 gpadmin 的密码 。
新的 密码：
无效的密码： 密码少于 8 个字符
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[root@gpmaster md10]#

修改/etc/hosts文件，注意这里有点很重要，因为搭建这个环境在后面初始化的时候一直报错，这里建议把下面127.0.0.1和::1前两行给注释掉。

[gpadmin@gpmaster ~]$ cat /etc/hosts
###127.0.0.1   localhost gpmaster localhost4 localhost4.localdomain4
###::1         localhost gpmaster localhost6 localhost6.localdomain6

192.168.223.150 gpmaster
192.168.223.151 gpslave1
192.168.223.152 gpslave2
192.168.223.153 gpslave3
[gpadmin@gpmaster ~]$

安装数据库软件

先去Greenplum官网下载安装包，地址：https://network.pivotal.io/products/vmware-tanzu-greenplum#/releases/158026/file_groups/1083。有两种安装形式：RPM和BINARY编译安装。RPM安装包我试过，rpm -ivh安装好像没有提示安装路径，默认路径安装到/usr/local/greenplum下面。rpm -ivh ./greenplum-db-5.10.2-rhel7-x86_64.rpm一键安装。

第二种是用源码文件安装，需要下载上图倒数第二个，下载下来是个zip安装包，然后解压到home目录。找到.bin文件，直接执行安装。

[root@gpslave1 ~]# yum install zip
已加载插件：fastestmirror
Determining fastest mirrors
 * base: mirrors.aliyun.com
 * extras: mirrors.neusoft.edu.cn
 * updates: mirrors.aliyun.com
base                                                                                | 3.6 kB  00:00:00     
extras                                                                              | 2.9 kB  00:00:00     
updates                                                                             | 2.9 kB  00:00:00     
正在解决依赖关系
--> 正在检查事务
---> 软件包 zip.x86_64.0.3.0-11.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

===========================================================================================================
 Package               架构                     版本                          源                      大小
===========================================================================================================
正在安装:
 zip                   x86_64                   3.0-11.el7                    base                   260 k

事务概要
===========================================================================================================
安装  1 软件包

总下载量：260 k
安装大小：796 k
Is this ok [y/d/N]: y
Downloading packages:
zip-3.0-11.el7.x86_64.rpm                                                           | 260 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : zip-3.0-11.el7.x86_64                                                                  1/1 
  验证中      : zip-3.0-11.el7.x86_64                                                                  1/1 

已安装:
  zip.x86_64 0:3.0-11.el7                                                                                  

完毕！
[root@gpslave1 ~]# cp /mnt/hgfs/shared/greenplum-db-5.10.2-rhel7-x86_64.zip /home/
[root@gpslave1 ~]# cd /home/
[root@gpslave1 home]# ll
总用量 191228
drwxr-xr-x. 2 root    root            6 9月   1 23:14 cdrom
-rwxrwxrwx. 1 root    root           33 9月   2 17:45 cmd
drwx------. 2 gpadmin gpadmin        62 9月  28 23:29 gpadmin
-rwxr-xr-x. 1 root    root    195802895 10月  9 02:01 greenplum-db-5.10.2-rhel7-x86_64.zip
drwxr-xr-x. 2 root    root           76 9月   4 00:55 gz_test
-rw-r--r--. 1 root    root          112 9月   4 00:43 gz_test.tar.gz
-rwxrwxrwx. 1 root    root          113 9月   3 16:37 tar_test.tar
-rwxrwxrwx. 1 root    root            0 9月   2 17:44 test.txt
drwxr-xr-x. 3 root    root           34 9月   1 23:32 VMTools
drwxr-xr-x. 9 root    root          175 11月 11 2015 vmware-tools-distrib
[root@gpslave1 home]# unzip greenplum-db-5.10.2-rhel7-x86_64.zip
-bash: unzip: 未找到命令
[root@gpslave1 home]# yum install unzip
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.neusoft.edu.cn
 * updates: mirrors.aliyun.com
正在解决依赖关系
--> 正在检查事务
---> 软件包 unzip.x86_64.0.6.0-22.el7_9 将被 安装
--> 解决依赖关系完成

依赖关系解决

===========================================================================================================
 Package               架构                   版本                           源                       大小
===========================================================================================================
正在安装:
 unzip                 x86_64                 6.0-22.el7_9                   updates                 171 k

事务概要
===========================================================================================================
安装  1 软件包

总下载量：171 k
安装大小：365 k
Is this ok [y/d/N]: y
Downloading packages:
unzip-6.0-22.el7_9.x86_64.rpm                                                       | 171 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : unzip-6.0-22.el7_9.x86_64                                                              1/1 
  验证中      : unzip-6.0-22.el7_9.x86_64                                                              1/1 

已安装:
  unzip.x86_64 0:6.0-22.el7_9                                                                              

完毕！
[root@gpslave1 home]# unzip greenplum-db-5.10.2-rhel7-x86_64.zip
Archive:  greenplum-db-5.10.2-rhel7-x86_64.zip
  inflating: greenplum-db-5.10.2-rhel7-x86_64.bin  
[root@gpslave1 home]# ll
总用量 384496
drwxr-xr-x. 2 root    root            6 9月   1 23:14 cdrom
-rwxrwxrwx. 1 root    root           33 9月   2 17:45 cmd
drwx------. 2 gpadmin gpadmin        62 9月  28 23:29 gpadmin
-rwxr-xr-x. 1 root    root    197905185 8月  10 2018 greenplum-db-5.10.2-rhel7-x86_64.bin
-rwxr-xr-x. 1 root    root    195802895 10月  9 02:01 greenplum-db-5.10.2-rhel7-x86_64.zip
drwxr-xr-x. 2 root    root           76 9月   4 00:55 gz_test
-rw-r--r--. 1 root    root          112 9月   4 00:43 gz_test.tar.gz
-rwxrwxrwx. 1 root    root          113 9月   3 16:37 tar_test.tar
-rwxrwxrwx. 1 root    root            0 9月   2 17:44 test.txt
drwxr-xr-x. 3 root    root           34 9月   1 23:32 VMTools
drwxr-xr-x. 9 root    root          175 11月 11 2015 vmware-tools-distrib
[root@gpslave1 home]# ./greenplum-db-5.10.2-rhel7-x86_64.bin 

********************************************************************************
You must read and accept the Pivotal Database license agreement
before installing
********************************************************************************

       ***  IMPORTANT INFORMATION - PLEASE READ CAREFULLY  ***

PIVOTAL GREENPLUM DATABASE END USER LICENSE AGREEMENT

IMPORTANT - READ CAREFULLY: This Software contains computer programs and
other proprietary material and information, the use of which is subject to
and expressly conditioned upon acceptance of this End User License
Agreement ("EULA").

This EULA is a legally binding document between you (meaning the person or
the entity that obtained the Software under the terms and conditions of
this EULA, is agreeing to be bound by the terms and conditions of this
EULA, and is referred to below as "You" or "Customer") and Pivotal (meaning
(i) Pivotal Software, Inc., if Customer is located in the United States;
and (ii) the local Pivotal sales subsidiary, if Customer is located in a
country outside the United States in which Pivotal has a local sales
subsidiary; and (iii) GoPivotal International Limited, if Customer is
located in a country outside the United States in which Pivotal does not
have a local sales subsidiary (in each case, referred to herein as
"Pivotal"). Unless Customer has entered into a written and separately
signed agreement with Pivotal that is currently in effect with respect to
the license of the Software and provision of Support Services and
Subscription Services, this EULA governs Customer's use of the Software and
the provision of Support Services and Subscription Services. Capitalized

前面都是一些开源软件的条款，直接按空格跳过。来到最重要的安装目录，因为我安装master的时候采用了默认安装路径，这里也保持一致，默认安装。

********************************************************************************
Do you accept the Pivotal Database license agreement? [yes|no]
********************************************************************************

 y

********************************************************************************
Provide the installation path for Greenplum Database or press ENTER to 
accept the default installation path: /usr/local/greenplum-db-5.10.2
********************************************************************************

********************************************************************************
/usr/local/greenplum-db-5.10.2 does not exist.
Create /usr/local/greenplum-db-5.10.2 ? [yes|no]
(Selecting no will exit the installer)
********************************************************************************

Extracting product to /usr/local/greenplum-db-5.10.2

********************************************************************************
Installation complete.
Greenplum Database is installed in /usr/local/greenplum-db-5.10.2

Pivotal Greenplum documentation is available
for download at http://gpdb.docs.pivotal.io
********************************************************************************

安装完后把greenplum的环境变量应用到当前环境，用source 命令执行greenplum安装目录下的shell脚本（所有机器，包括master和slave）

[root@gpslave1 home]# source /usr/local/greenplum-db/greenplum_path.sh

greenplum_path.sh 这个文件设置了greenplum集群环境需要的各种环境变量，如果不执行source /usr/local/greenplum-db/greenplum_path.sh，管理greenplum所需要的各种命令（累：gpstart，gpstate）会因为没有设置环境变量而找不到，无法执行。

[gpadmin@gpmaster ~]$ cat /usr/local/greenplum-db/greenplum_path.sh
GPHOME=/usr/local/greenplum-db-5.10.2

# Replace with symlink path if it is present and correct
if [ -h ${GPHOME}/../greenplum-db ]; then
    GPHOME_BY_SYMLINK=`(cd ${GPHOME}/../greenplum-db/ && pwd -P)`
    if [ x"${GPHOME_BY_SYMLINK}" = x"${GPHOME}" ]; then
        GPHOME=`(cd ${GPHOME}/../greenplum-db/ && pwd -L)`/.
    fi
    unset GPHOME_BY_SYMLINK
fi
#setup PYTHONHOME
if [ -x $GPHOME/ext/python/bin/python ]; then
    PYTHONHOME="$GPHOME/ext/python"
fi
PYTHONPATH=$GPHOME/lib/python
PATH=$GPHOME/bin:$PYTHONHOME/bin:$PATH
LD_LIBRARY_PATH=$GPHOME/lib:$PYTHONHOME/lib:${LD_LIBRARY_PATH-}
export LD_LIBRARY_PATH
OPENSSL_CONF=$GPHOME/etc/openssl.cnf
export GPHOME
export PATH
export PYTHONPATH
export PYTHONHOME
export OPENSSL_CONF
[gpadmin@gpmaster ~]$

然后修改gpadmin主目录下的.bash_profle和.bashrc两个用户配置文件（所有机器，包括master和slave）

.bash_profle加入下面4条命令，第一条是为了开机可以加载GP的环境变量，gpadmin用户可以调用GP的命令，第二条设置主数据库的数据文件目录，安装集群的时候这个很关键，第3条是GP的数据库端口号。

source /usr/local/greenplumn-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/gpmaster/gpseg-1
export PGPORT=5432
export PGDATABASE=testDB

.bashrc加入下面命令（所有机器，包括master和slave）

MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/gpmaster/gpseg-1
export MASTER_DATA_DIRECTORY
source /usr/local/greenplum-db/greenplum_path.sh

关闭防火墙（所有机器，包括master和slave）

[gpadmin@gpmaster ~]$ systemctl stop firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password: 
==== AUTHENTICATION COMPLETE ===
[gpadmin@gpmaster ~]$ systemctl disable firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-unit-files ===
Authentication is required to manage system service or unit files.
Authenticating as: root
Password: 
==== AUTHENTICATION COMPLETE ===
==== AUTHENTICATING FOR org.freedesktop.systemd1.reload-daemon ===
Authentication is required to reload the systemd state.
Authenticating as: root
Password: 
==== AUTHENTICATION COMPLETE ===
[gpadmin@gpmaster ~]$

关闭selinux（所有机器，包括master和slave）

[gpadmin@gpmaster ~]$ cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 


[gpadmin@gpmaster ~]$

修改OS内核参数（所有机器，包括master和slave）

[gpadmin@gpmaster ~]$ cat /etc/sysctl.conf 
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#
#


kernel.shmmax = 1800000000000
kernel.shmmni = 8192
kernel.shmall = 1800000000000
kernel.sem = 1000 10240000 400 10240
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.overcommit_memory = 2
vm.swappiness = 1
kernel.pid_max = 655360
[gpadmin@gpmaster ~]$ sysctl -p

修改完成记得sysctl -p应用下修改生效

修改资源限制参数（所有机器，包括master和slave）

修改/etc/security/limits.conf，加入下面4行

* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072

同步集群的时钟（所有机器，包括master和slave），yum install ntp安装一下NTP，再用date看下几台服务器的时间是否校正过来了

[gpadmin@gpmaster ~]$ gpssh -f hostlist 
=> date
[gpmaster] 2021年 10月 15日 星期五 22:44:58 CST
[gpslave2] 2021年 10月 15日 星期五 22:44:58 CST
[gpslave1] 2021年 10月 15日 星期五 22:44:58 CST
[gpslave3] 2021年 10月 15日 星期五 22:44:58 CST
=>

（所有机器，包括master和slave）Greenplum数据库管理程序中的gpexpand‘ gpinitsystem、gpaddmirrors，使用 SSH连接来执行任务。在规模较大的Greenplum集群中，程序的ssh连接数可能会超出主机的未认证连接的最大阈值。发生这种情况时，会收到以下错误：ssh_exchange_identification：连接被远程主机关闭。为避免这种情况，可以更新 /etc/sshd_config 文件的 MaxStartups 和 MaxSessions 参数，前者是最大允许并发连接数，后者是单个连接最大允许会话数。一般改前者就够了。

[root@gpmaster gpadmin]# vim /etc/ssh/sshd_config 
[root@gpmaster gpadmin]# cat /etc/ssh/sshd_config | grep "MaxStartups"
MaxStartups 300:30:1000
[root@gpmaster gpadmin]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service
[root@gpmaster gpadmin]#

接下来配置greenplum集群环境的服务器列表和segment列表，我们要先准备两个文件，hostlist和seg_hosts。然后生成SSH KEY让几台服务器可以无需每次输入密码自动登录通讯。

[root@gpmaster home]# cd /home/gpadmin/
[root@gpmaster gpadmin]# touch hostlist
[root@gpmaster gpadmin]# vim hostlist
[root@gpmaster gpadmin]# touch seg_hosts
[root@gpmaster gpadmin]# vim seg_hosts
[root@gpmaster gpadmin]# cat hostlist
gpmaster
gpslave1
gpslave2
gpslave3
[root@gpmaster gpadmin]# cat seg_hosts
gpslave1
gpslave2
gpslave3

用gpssh-exkeys命令打通master和slave机器，免密登录并执行命令

[root@gpmaster gpadmin]# su gpadmin
[gpadmin@gpmaster ~]$ gpssh-exkeys -f hostlist
[STEP 1 of 5] create local ID and authorize on local host
  ... /home/gpadmin/.ssh/id_rsa file exists ... key generation skipped

[STEP 2 of 5] keyscan all hosts and update known_hosts file

[STEP 3 of 5] authorize current user on remote hosts
  ... send to gpslave1
  ... send to gpslave2
  ... send to gpslave3

[STEP 4 of 5] determine common authentication file content

[STEP 5 of 5] copy authentication files to all remote hosts
  ... finished key exchange with gpslave1
  ... finished key exchange with gpslave2
  ... finished key exchange with gpslave3

[INFO] completed successfully
[gpadmin@gpmaster ~]$

然后用gpscp或者scp把这hostlist 和seg_hosts文件拷贝到其他的机器

Greenplum的数据分为主数据和用户数据库数据（用户数据库数据又有镜像数据），所以我们要分别创建不同的目录存储。gpmaster是存主数据的，gpdatap1和gpdatap2都是用户数据库数据的Primary节点，gpdatam1和gpdatam2是Mirror镜像数据节点。

[root@gpmaster gpadmin]# gpssh -f hostlist 
=> mkdir -p /home/gpadmin/gpdata/gpmaster
[gpmaster]
[gpslave1]
[gpslave3]
[gpslave2]
=> mkdir -p /home/gpadmin/gpdata/gpdatap1
[gpmaster]
[gpslave1]
[gpslave3]
[gpslave2]
=> mkdir -p /home/gpadmin/gpdata/gpdatap2
[gpmaster]
[gpslave1]
[gpslave3]
[gpslave2]
=> mkdir -p /home/gpadmin/gpdata/gpdatam1
[gpmaster]
[gpslave1]
[gpslave3]
[gpslave2]
=> mkdir -p /home/gpadmin/gpdata/gpdatam2
[gpmaster]
[gpslave1]
[gpslave3]
[gpslave2]
=>

这里记得chown -R gpadmin:gpadmin /home/gpadmin，把几台服务器的数据目录的权限给到gpadmin，因为等下集群安装的时候是gpadmin的身份安装的，要求gpadmin用户对目录有绝对控制权，不然安装过程会报错权限问题

=> chown -R gpadmin:gpadmin /home/gpadmin
[gpslave1]
[gpslave3]
[gpslave2]

前面的准备工作都做完，接下来就是集群初始化，先生成一个初始化配置文件，greenplum安装目录下提供一个模板可以修改，先拷贝到主目录下

[gpadmin@gpmaster ~]$ cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config ~/init_conf

这里我是先添加primary，先不添加mirror和standby，这两者可以单独通过gpaddmirrors和gpinitstandby单独添加。

[gpadmin@gpmaster ~]$ cat init_conf
# FILE NAME: gpinitsystem_config

# Configuration file needed by the gpinitsystem

################################################
#### REQUIRED PARAMETERS
################################################

#### Name of this Greenplum system enclosed in quotes.
ARRAY_NAME="Greenplum Data Platform"
MACHINE_LIST_FILE=/home/gpadmin/seg_hosts

#### Naming convention for utility-generated data directories.
SEG_PREFIX=gpseg

#### Base number by which primary segment port numbers
#### are calculated.

PORT_BASE=33000

#### File system location(s) where primary segment data directories
#### will be created. The number of locations in the list dictate
#### the number of primary segments that will get created per
#### physical host (if multiple addresses for a host are listed in
#### the hostfile, the number of segments will be spread evenly across
#### the specified interface addresses).
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatap1 /home/gpadmin/gpdata/gpdatap2)

#### OS-configured hostname or IP address of the master host.
MASTER_HOSTNAME=gpmaster

#### File system location where the master data directory
#### will be created.
MASTER_DIRECTORY=/home/gpadmin/gpdata/gpmaster

#### Port number for the master instance.
MASTER_PORT=5432

#### Shell utility used to connect to remote hosts.
TRUSTED_SHELL=/usr/bin/ssh

#### Maximum log file segments between automatic WAL checkpoints.
CHECK_POINT_SEGMENTS=8

#### Default server-side character set encoding.
ENCODING=UNICODE

################################################
#### OPTIONAL MIRROR PARAMETERS
################################################

#### Base number by which mirror segment port numbers
#### are calculated.
#MIRROR_PORT_BASE=43000

#### Base number by which primary file replication port
#### numbers are calculated.
#REPLICATION_PORT_BASE=34000

#### Base number by which mirror file replication port
#### numbers are calculated.
#MIRROR_REPLICATION_PORT_BASE=44000

#### File system location(s) where mirror segment data directories
#### will be created. The number of mirror locations must equal the
#### number of primary locations as specified in the
#### DATA_DIRECTORY parameter.
#declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatam1 /home/gpadmin/gpdata/gpdatam2)

################################################
#### OTHER OPTIONAL PARAMETERS
################################################

#### Create a database of this name after initialization.
#DATABASE_NAME=name_of_database

#### Specify the location of the host address file here instead of
#### with the the -h option of gpinitsystem.
#MACHINE_LIST_FILE=/home/gpadmin/gpconfigs/hostfile_gpinitsystem

初始化之前可以用命令检查下几台服务器的网络连通性和性能

[gpadmin@gpmaster ~]$ gpcheckperf -f /home/gpadmin/hostlist -r N -d /tmp
/usr/local/greenplum-db/./bin/gpcheckperf -f /home/gpadmin/hostlist -r N -d /tmp

-------------------
--  NETPERF TEST
-------------------

====================
==  RESULT
====================
Netperf bisection bandwidth test
gpmaster -> gpslave1 = 116.070000
gpslave2 -> gpslave3 = 139.770000
gpslave1 -> gpmaster = 549.060000
gpslave3 -> gpslave2 = 469.300000

Summary:
sum = 1274.20 MB/sec
min = 116.07 MB/sec
max = 549.06 MB/sec
avg = 318.55 MB/sec
median = 469.30 MB/sec

添加完就执行 gpinitsystem -c ~/init_conf -a -h hostlist，这个过程会有提示，输入y继续就行。正常初始化不会遇到什么问题。会遇到的问题无非就是前面提到的权限（所以要chown -R gpadmn:gpadmin /home/gpadmin把数据目录的控制权给安装用户），还有可能遇到就是安装目录下面的primary、master目录不为空，再者就是机器间网络问题，ping不同。但是前面关闭了selinux、防火墙后这个问题应该不存在。还有可能的问题就是端口被占用，就是前面在.bash_profile里面配置的pgport，这个一般是第一次初始化的时候报错，重新初始化的时候会出现，一般会提示端口被占用或者/tmp目录下某个.lock文件存在（端口进程文件）。这里需要先用lsof -i:端口号查看下是否有进程在使用这个端口，可以kill -9 Pid杀死进程（这个命令默认也是不安装的，需要yum install lsof安装）。至于.lock文件，rm -rf 文件路径删除就是了。

这里说一下重装，我搭设这个环境几乎花了2 3天时间，过程中各种报错重装，光重新初始化就不知道多少次。每次如果安装完失败了，重新初始化之前需要做哪些事情：1、gpssh -f 登录集群环境机器后，rm -rf 删除所有机器的maste、primary、mirror（如果有）的数据目录下所有的文件和目录； 2、 lsof -i: PGPORT查看一下GP的端口是否有进程占用，有的话Kill -9杀死进程。然后重新跑一下初始化命令，如果提示.lock文件存在，就rm -rf移除，然后就可以顺利重新初始化了。

初始化过程结束后，先用gpstate -s检查下segment的同步情况。

然后再gpaddmirrors 生成添加镜像的配置文件，过程中你需要输入镜像目录的地址

[gpadmin@gpmaster ~]$ gpaddmirrors -o ~/addmirror

然后再相同命令，改成-i ~/addmirror执行添加镜像

[gpadmin@gpmaster ~]$ gpaddmirrors -i ~/addmirror

顺利的话会出现sucessfully的提示。

gpstate -m查看镜像的同步情况，输出的结果如果提示Data Status=Synchronized镜像同步没有问题。

[gpadmin@gpmaster ~]$ gpstate -m
20211016:00:42:28:063483 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -m
20211016:00:42:33:063483 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:09'
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master...
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--Current GPDB mirror list and status
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--Type = Group
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   Mirror     Datadir                                Port    Status    Data Status    
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave1   /home/gpadmin/gpdata/gpdatam1/gpseg0   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave1   /home/gpadmin/gpdata/gpdatam2/gpseg1   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave2   /home/gpadmin/gpdata/gpdatam1/gpseg2   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave2   /home/gpadmin/gpdata/gpdatam2/gpseg3   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave3   /home/gpadmin/gpdata/gpdatam1/gpseg4   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave3   /home/gpadmin/gpdata/gpdatam2/gpseg5   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpmaster   /home/gpadmin/gpdata/gpdatam1/gpseg6   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpmaster   /home/gpadmin/gpdata/gpdatam2/gpseg7   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------

然后再用gpinitstandby - s gpslave3 添加standby节点。

至此环境搭建就结束了。

gpstate -s查看整体的同步情况。仔细观察master、primary和mirror下面每个segment的同步和状态，不要出现Fatal或者Error的字眼

[gpadmin@gpmaster ~]$ gpstate -m
20211016:00:42:28:063483 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -m
20211016:00:42:33:063483 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:09'
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master...
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--Current GPDB mirror list and status
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--Type = Group
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   Mirror     Datadir                                Port    Status    Data Status    
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave1   /home/gpadmin/gpdata/gpdatam1/gpseg0   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave1   /home/gpadmin/gpdata/gpdatam2/gpseg1   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave2   /home/gpadmin/gpdata/gpdatam1/gpseg2   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave2   /home/gpadmin/gpdata/gpdatam2/gpseg3   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave3   /home/gpadmin/gpdata/gpdatam1/gpseg4   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpslave3   /home/gpadmin/gpdata/gpdatam2/gpseg5   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpmaster   /home/gpadmin/gpdata/gpdatam1/gpseg6   34000   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:-   gpmaster   /home/gpadmin/gpdata/gpdatam2/gpseg7   34001   Passive   Synchronized
20211016:00:42:34:063483 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
[gpadmin@gpmaster ~]$ gpstate -s
20211016:00:44:35:063553 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -s
20211016:00:44:36:063553 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20211016:00:44:36:063553 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:09'
20211016:00:44:36:063553 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master...
20211016:00:44:43:063553 gpstate:gpmaster:gpadmin-[INFO]:-Gathering data from segments...
................ 
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:--Master Configuration & Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master host                    = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master postgres process ID     = 36018
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master data directory          = /home/gpadmin/gpdata/gpmaster/gpseg-1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master port                    = 5432
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master current role            = dispatch
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Greenplum initsystem version   = 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Greenplum current version      = PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:09
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Postgres version               = 8.3.23
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Master standby                 = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Standby master state           = Standby host passive
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-Segment Instance Status Report
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap1/gpseg0
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 35980
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam1/gpseg0
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 10300
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap2/gpseg1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 35981
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam2/gpseg1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 10299
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap1/gpseg2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9866
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam1/gpseg2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9842
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave1
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap2/gpseg3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9865
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam2/gpseg3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9845
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap1/gpseg4
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9411
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam1/gpseg4
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9906
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave2
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap2/gpseg5
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9410
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam2/gpseg5
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9905
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap1/gpseg6
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9471
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam1/gpseg6
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34000
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 36584
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpslave3
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatap2/gpseg7
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 33001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Primary
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 9472
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Database status                   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Segment Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Hostname                          = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Address                           = gpmaster
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/gpdata/gpdatam2/gpseg7
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Port                              = 34001
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Mirroring Info
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Current role                      = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Preferred role                    = Mirror
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Mirror status                     = Synchronized
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-   Status
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      PID                               = 36585
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Configuration reports status as   = Up
20211016:00:44:59:063553 gpstate:gpmaster:gpadmin-[INFO]:-      Segment status                    = Up

尝试连接一下数据库

[gpadmin@gpmaster ~]$ psql -d postgres
psql (8.3.23)
Type "help" for help.

postgres=#

看似很简单，我做之前也觉得挺简单的，但是做起来才发现遇到很多问题，这些问题的排错过程也不像Windows下安装SQL Server那么简单，这里只能是说总结一下过程的心得：

首先排错的话有几个地方：1、命令行的错误信息输出；2、安装目录下的gpAdminLogs目录下的log文件，每个gp命令都有一个对应的log日志，当然我发现这些日志跟命令行前端的信息输出基本一致。3、这点很重要，如果前面日志给到的信息没办法直接找到原因，那就只能根据信息提示发生错误的服务器的segment节点下面的pg_log目录去找到startup.log日志，以及相同目录下面的csv文件去排错了。后者的输出日志信息是最详尽的，我理解就是底层接口调用的日志输出。

但是既然如此，有这么多日志记录的地方，不见得排错就轻松。像我搭设环境的过程中遇到master成功，primary成功，但是mirror失败的情况，排错过程非常痛苦，提示错误信息没办法给到我直接了当的错误问题提示，只是在初始化成功后commencing primary and mirror segment这段报错，提示mirror启动失败，提示 REASON: Conversion Fail。这个信息google上也找不到太多有用的信息。后来我自己通过两天的google，总算找到比较有用的几篇比较有用的文章。

这个排错的经历让我有比较深的感触：

1、日志多有时候并不意味着就非常好，因为带有错误信息提示的日志越多越容易混淆你对根源错误的判断。

2、解决问题最终还是要多利用计算机基础知识，以及把复杂的步骤拆解程多个简单的步骤做。我在初始化的时候直接把primary、mirror和standby分成三步来做（这篇文章给了我启发 https://github.com/greenplum-db/gpdb/issues/495）。在添加mirror的时候成功添加了mirror，也可以看到mirror目录下成功创建了文件目录，去到mirror的pg_log目录下找到csv文件看，找到有WARINING的日志错误提示，提示是primary/mirror数据同步出现问题，但是mirror其实是创建成功了，通过gpstate -s也可以看到mirror已经添加成功了，只是状态是 Out of sync。

PostmasterPrimaryMirrorTransition (2) Finished with Error
'run consumer', mirroring role 'primary role' mirroring state 'resync' segment state 'transition to resync' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(2248)' 'cdbfilerepprimaryack.c' 'L957' 'FileRepAckPrimary_RunConsumer'
PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InResync resulted in Error
failure is detected in segment mirroring, failover requested
received transition request packet. processing the request
PrimaryMirrorTransitionRequest: to primary/mirror mode PrimarySegment, data state InChangeTracking, host gpslave1, port 36000, peer gpslave2, peerPort 35000
TransitiontoPrimary: shutting down filerep backends
TransitiontoPrimary: waiting for filerep backends to shutdown
'process exit, process name 'primary resync manager process' process pid '2250' exit status '0' ', mirroring role 'primary role' mirroring state 'resync' segment state 'in backends shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(2244)' 'cdbfilerep.c' 'L2114' 'LogChildExit'

但上面的错误提示其实还是给不了太多的有用的信息。我用另外的方式来处理，先用gprecoverseg看有哪些segment需要修复。 -o ./recov 和

这个时候我在gp的官方社区找到一篇KB非常有启发。里面告诉我们先通过 segment的日志去看问题，再nc命令去排除网络和端口无法访问的问题（这就是我说了回归到计算机的基础知识来看问题，先确保上面提示的mirror端口 36000在gpslave1上是可以被其他机器访问的。）。所以我用lsof命令和nc命令分别对本地的端口监听情况和访问情况测试。确定端口是启用的。

nc -v -z gpslave1 36000

https://community.pivotal.io/s/article/Down-segments-not-coming-up-online?language=en_US

最后我是怎么解决问题的。gprecoverseg命令提示了我127.0.0.1失败，最后我才想起来，我把master上面的hosts文件拷贝到其他机器，导致其他机器把127.0.0.1指向了master。我赶紧把127.0.0.1和::1这两行注释掉。然后再执行gpstop 和 gpstart。结果mirror恢复了同步。

参考：

安装教程：

https://blog.csdn.net/xyy430104/article/details/83089988

https://blog.csdn.net/q936889811/article/details/85603814?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link

https://www.cnblogs.com/jnba/p/10528220.html

https://www.cnblogs.com/lottu/p/8717400.html

https://www.cnblogs.com/sccq-1/p/14832185.html

https://www.cnblogs.com/langke93/p/6517159.html

https://www.cnblogs.com/scoter2008/p/5928972.html

https://chowdera.com/2021/07/20210728084533317j.html

https://blog.csdn.net/luozhenjie123/article/details/112645119

命令教学：

https://blog.csdn.net/qq_43303221/article/details/88800025

https://blog.51cto.com/michaelkang/2166368

https://blog.csdn.net/kjh2007abc/article/details/85000753

排错：

http://blog.itpub.net/31439444/viewspace-2699576/

https://blog.csdn.net/q936889811/article/details/85612046

https://blog.csdn.net/jwocnimabi/article/details/106864905

https://www.codercto.com/a/20728.html

http://blog.itpub.net/31439444/viewspace-2699576/

https://github.com/greenplum-db/gpdb/issues/495

https://community.pivotal.io/s/article/Down-segments-not-coming-up-online?language=en_US

https://groups.google.com/a/greenplum.org/g/gpdb-users/c/B5qTiF0KWRI

心得：

https://cloud.tencent.com/developer/article/1800233

https://blog.csdn.net/qq_42979842/article/details/108553571

相关阅读:
ride.py 启动报错
 python中的__call__函数
 python-namedtuple命名元组
 python模块-操作excel（xlsxwriter）
（转）python中@property详解
 python中使用logging将日志写入文件或输出到控制台
 “selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities“解决办法
 Fiddler对https抓包时，提示"HTTPS decryption is disabled."
fiddler的使用
 常用软件激活/破解
原文地址：https://www.cnblogs.com/jenrrychen/p/15271312.html