在kubernetes集群里集成Apollo配置中心(3)之交付Apollo-portal至Kubernetes集群

1.执行apollo-portal数据库脚本

apollo-portal数据库脚本链接：https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql

在mfyxw10.mfyxw.com主机上操作

(1)将apollo-portal数据库脚本下载回来

[root@mfyxw10 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql -O apolloportal.sql

(2)导入apollo-portal脚本

[root@mfyxw10 ~]# mysql -uroot -p < apolloportal.sql

(3)查看是否导入成功

[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> show databases;          #Apollo-portal的数据库名为：ApolloPortalDB
+--------------------+
| Database           |
+--------------------+
| ApolloConfigDB     |
| ApolloPortalDB     |
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)

MariaDB [(none)]> use ApolloPortalDB;      #切换至ApolloPortalDB数据库
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [ApolloPortalDB]> show tables;    #查看ApolloPortalDB数据库下的数据表
+--------------------------+
| Tables_in_ApolloPortalDB |
+--------------------------+
| App                      |
| AppNamespace             |
| Authorities              |
| Consumer                 |
| ConsumerAudit            |
| ConsumerRole             |
| ConsumerToken            |
| Favorite                 |
| Permission               |
| Role                     |
| RolePermission           |
| ServerConfig             |
| UserRole                 |
| Users                    |
+--------------------------+
14 rows in set (0.00 sec)

MariaDB [ApolloPortalDB]> 

(4)ApolloPortal数据库授权

[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"192.168.80.%" identified by "123456";
MariaDB [(none)]>flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> 

(5)更新ApolloPortal数据表ServerConfig的Value

[root@mfyxw10 ~]# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or g.
Your MariaDB connection id is 44
Server version: 10.1.45-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB [(none)]> use ApolloPortalDB;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [ApolloPortalDB]> show tables;
+--------------------------+
| Tables_in_ApolloPortalDB |
+--------------------------+
| App                      |
| AppNamespace             |
| Authorities              |
| Consumer                 |
| ConsumerAudit            |
| ConsumerRole             |
| ConsumerToken            |
| Favorite                 |
| Permission               |
| Role                     |
| RolePermission           |
| ServerConfig             |
| UserRole                 |
| Users                    |
+--------------------------+
14 rows in set (0.01 sec)

MariaDB [ApolloPortalDB]> select * from ServerConfigG;
*************************** 1. row ***************************
                       Id: 1
                      Key: apollo.portal.envs
                    Value: dev
                  Comment: 可支持的环境列表
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 2. row ***************************
                       Id: 2
                      Key: organizations
                    Value: [{"orgId":"TEST1","orgName":"样例部门1"},{"orgId":"TEST2","orgName":"样例部门2"}]
                  Comment: 部门列表
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 3. row ***************************
                       Id: 3
                      Key: superAdmin
                    Value: apollo
                  Comment: Portal超级管理员
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 4. row ***************************
                       Id: 4
                      Key: api.readTimeout
                    Value: 10000
                  Comment: http接口read timeout
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 5. row ***************************
                       Id: 5
                      Key: consumer.token.salt
                    Value: someSalt
                  Comment: consumer token salt
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 6. row ***************************
                       Id: 6
                      Key: admin.createPrivateNamespace.switch
                    Value: true
                  Comment: 是否允许项目管理员创建私有namespace
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
*************************** 7. row ***************************
                       Id: 7
                      Key: configView.memberOnly.envs
                    Value: pro
                  Comment: 只对项目成员显示配置信息的环境列表，多个env以英文逗号分隔
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2020-07-06 10:15:59
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2020-07-06 10:15:59
7 rows in set (0.00 sec)

ERROR: No query specified

MariaDB [ApolloPortalDB]> 

更新前的内容如图片所示

更新后如下图所示

[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> use ApolloPortalDB;
MariaDB [ApolloPortalDB]> update ServerConfig set Value = '[{"orgId":"yf1","orgName":"Linux学院"},{"orgId":"yf2","orgName":"云计算学院"},{"orgId":"yf3","orgName":"Python学院"},{"orgId":"yf4","orgName":"大数据学院"}]' where Id = 2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

2.准备apollo-portal软件

在运维主机mfyxw50.mfyxw.com上操作

(1)下载apollo-portal软件

[root@mfyxw50 ~]# cd /opt/src/
[root@mfyxw50 ~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip

(2)解压apollo-portal软件至相对应的目录

[root@mfyxw50 ~]# mkdir /data/dockerfile/apollo-portal
[root@mfyxw50 ~]# unzip /opt/src/apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal

(3)删除apollo-portal目录用不到的文件

[root@mfyxw50 ~]# rm -fr /data/dockerfile/apollo-portal/apollo-portal-1.5.1-sources.jar
[root@mfyxw50 ~]# rm -fr /data/dockerfile/apollo-portal/apollo-portal.conf
[root@mfyxw50 ~]# rm -fr /data/dockerfile/apollo-portal/scripts/shutdown.sh

(4)配置数据库连接串

[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-portal/config/application-github.properties << EOF
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
EOF

(5)配置Portal的mate service(此处可以不修改，在configmap处已经引入了)

[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-portal/config/apollo-env.properties << EOF
local.meta=http://localhost:8080
dev.meta=http://config.od.com:8080
fat.meta=http://fill-in-fat-meta-server:8080
uat.meta=http://fill-in-uat-meta-server:8080
lpt.meta=${lpt_meta}
pro.meta=http://fill-in-pro-meta-server:8080
EOF

(6)更新apollo-portal的startup.sh文件

[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-portal/scripts/startup.sh << EOF
#!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== 
"

cd \`dirname $0\`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null
EOF

3.制作Apollo-Portal的Docker镜像

在运维主机(mfyxw50.mfyxw.com)上操作

(1)编写Apollo-Portal的Docker文件

[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-portal/Dockerfile << EOF
FROM harbor.od.com/base/jre8:8u112

ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]
EOF

(2)制作Apollo-portal镜像

[root@mfyxw50 ~]# cd /data/dockerfile/apollo-portal
[root@mfyxw50 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
[root@mfyxw50 apollo-portal]# docker login harbor.od.com
[root@mfyxw50 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1

4.解析portal.od.com域名

在mfyxw10.mfyxw.com主机操作

(1)设置域名

[root@mfyxw10 ~]# cat > /var/named/od.com.zone << EOF
$ORIGIN od.com.
$TTL 600   ; 10 minutes
@       IN  SOA dns.od.com.   dnsadmin.od.com. (
                             ;序号请加1，表示比之前版本要新
                             2020031312 ; serial
                             10800          ; refresh (3 hours)
                             900              ; retry (15 minutes)
                             604800         ; expire (1 week)
                             86400          ; minimum (1 day)
                              )
                      NS   dns.od.com.
$TTL 60 ;  1 minute
dns             A          192.168.80.10
harbor          A          192.168.80.50   ;添加harbor记录
k8s-yaml        A          192.168.80.50
traefik         A          192.168.80.100
dashboard       A          192.168.80.100
zk1             A          192.168.80.10
zk2             A          192.168.80.20
zk3             A          192.168.80.30
jenkins         A          192.168.80.100
dubbo-monitor   A          192.168.80.100
demo            A          192.168.80.100
mysql           A          192.168.80.10
config          A          192.168.80.100
portal          A          192.168.80.100
EOF

(2)重启DNS服务

[root@mfyxw10 ~]# systemctl restart named

(3)测试解析

[root@mfyxw10 ~]# dig -t A portal.od.com @192.168.80.10 +short
192.168.80.100

5.准备Apollo-portal资源配置文件

在运维主机mfyxw50.mfyxw.com上操作

(1)创建用于存放Apollo-portal资源配置文件目录

[root@mfyxw50 ~]# mkdir -p /data/k8s-yaml/apollo-portal

(2)deployment.yaml文件内容如下：

[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-portal/deployment.yaml << EOF
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
EOF

(3)service.yaml文件内容如下：

[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-portal/service.yaml << EOF
kind: Service
apiVersion: v1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-portal
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None
EOF

(4)Ingress.yaml文件内容如下：

[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-portal/Ingress.yaml << EOF
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 8080
EOF

(5)ConfigMap.yaml文件内容如下：

[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-portal/configmap.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    dev.meta=http://config.od.com
EOF

6.应用Apollo-portal资源配置清单

在master节点(mfyxw30.mfyxw.com或mfyxw40.mfyxw.com)任意一台上执行即可

(1)应用Apollo-portal资源配置清单

[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/deployment.yaml
deployment.extensions/apollo-portal created
[root@mfyxw30 ~]# 
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/service.yaml
service/apollo-portal created
[root@mfyxw30 ~]# 
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/Ingress.yaml
ingress.extensions/apollo-portal created
[root@mfyxw30 ~]# 
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/configmap.yaml
configmap/apollo-portal-cm created

(2)查看Apollo-portal的pod是否正常启动

[root@mfyxw30 ~]# kubectl get pod -n infra
NAME                                   READY   STATUS    RESTARTS   AGE
apollo-adminservice-5cccf97c64-bhqzb   1/1     Running   1          6h22m
apollo-configservice-5f6555448-7wxsp   1/1     Running   1          6h16m
apollo-portal-57bc86966d-jz5vg         1/1     Running   0          112s
dubbo-monitor-6676dd74cc-9hghb         1/1     Running   13         17d
dubbo-monitor-6676dd74cc-rd86g         1/1     Running   12         17d
jenkins-b99776c69-p6skp                1/1     Running   17         39d

7.使用浏览器portal.od.com登录

Apollo-portal默认的用户为：apollo 密码为：admin 第一次登录后，强烈建议修改密码，在此修改密码为：admin123