一、配置中心概述
配置其实是独立于程序的可配变量，同一份程序在不同配置下会有不同的行为，常见的配置有连接字符串，应用配置和业务配置等。
1.配置的多种形态
- 程序内部hardcode，这种做法是反模式，一般我们不建议！
- 配置文件，比如spring应用成语的配置一般放在application.properties文件中
- 环境变量，配置可以预置在操作系统的环境变量里头，程序运行时读取。
- 启动参数，可以在程序启动时一次性提供参数，例如JAVA程序启动时可以通过java -D方式配启动参数。
- 基于数据库，有经验的开发人员会把易变的配置放在数据库中，这样可以在运行期灵活调整配置。
2.配置管理的痛点
- 配置散乱格式不标准（xml.ini,conf,yaml,json...）
- 主要采用本地静态配置，应用多副本集下配置修改麻烦
- 易引发事故（测试环境、生产环境配置混用）
- 配置缺乏安全审计和版本控制功能
- 不同环境的应用，配置不同，造成多次打包，测试失效
3.配置中心是什么
- 顾名思义，就是集中管理应用程序配置的中心
4.常见配置中心的种类
- XDiamond:全局配置中心，存储应用的配置项，解决配置混乱分散的问题。名字来源于淘宝的开源项目diamond，前面加一个字母x以示区别
- Qconf：它是一个分布式配置管理工具。用来替代传统的配置文件，使得配置信息和程序代码分离，同时配置变化能够实时同步到客户端，而且保证用户高效读取配置，这使得工程师从琐碎的配置修改，代码提交，配置上线程序中解放出来，极大的建华路配置管理工作。
- Disconf：专注于各种分布式系统配置管理的通用组件和通用平台，提供统一的配置管理服务
- SpringCloudConfig:它为分布式系统中的外部配置提供服务器和客户端支持。
- K8S ConfigMap：K8S的一种标准资源，专门用来集中管理应用的配置。
- Apollo：携程框架部门开源的，分布式配置中心。
二、实战k8s的配置中心-ConfigMap
1.环境初始化
~]# /opt/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
~]# ps aux | grep zoo
root 22252 0.0 0.0 112708 976 pts/0 S+ 16:51 0:00 grep --color=auto zoo
~]# cd /data/zookeeper/data/
data]# rm -rf ./*
data]# cd ../logs/
logs]# rm -rf ./*
logs]# vim /opt/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
logs]# /opt/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
logs]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
注：通过dashboard修改之前交付的dubbo三个deployment的pod期望值为0
2.拆分环境
| 主机名 | 角色 | ip |
|---|---|---|
| kjdow7-11.host.com | zk1.phc-dow.com(Test环境) | 10.4.7.11 |
| kjdow7-12.host.com | zk2.phc-dow.com(Prod环境) | 10.4.7.12 |
3.准备资源配置清单(dubbo-monitor)
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-monitor/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
dubbo.properties: |
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=kjdow
dubbo.registry.address=zookeeper://zk1.phc-dow.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
dubbo.log4j.level=WARN
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-monitor/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.phc-dow.com/infra/dubbo-monitor:latest
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
############################
volumeMounts:
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
volumes:
- name: configmap-volume
configMap:
name: dubbo-monitor-cm
############################
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
注： 跟之前的dp.yaml文件相比添加了标记内的几行
############################
通过configmap.yaml定义了一个名字为dubbo-monitor-cm的configmap资源
在dp.yaml中，首先定义了一个卷configmap-volume，内容为dubbo-monitor-cm（标记的最后一行，对应了configmap.yaml）,然后挂载卷configmap-volume，位置在/dubbo-monitor-simple/conf
############################
############################
[root@kjdow7-200 bin]# cat start.sh
#!/bin/bash
sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties
##可以看到在启动时会自动根据dubbo_origin.properties生成dubbo.properties，在这里通过configmap生成的dubbo.properties是只读的，因此启动这一行会报错可以忽略
3.1 自己根据已有的配置文件生成Configmap
[root@kjdow7-21 conf]# kubectl create cm kubelet-cm --from-file=./kubelet.kubeconfig
configmap/kubelet-cm created
[root@kjdow7-21 conf]# kubectl get cm kubelet-cm -o wide
NAME DATA AGE
kubelet-cm 1 15s
[root@kjdow7-21 conf]# kubectl get cm kubelet-cm -o yaml
4.应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/configmap.yaml
configmap/dubbo-monitor-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/dp.yaml
deployment.extensions/dubbo-monitor configured
如下图所示：节点已经启动
可以看到注册地址就是configmap上设置的
5 重新发版，修改dubbo项目的配置文件
打开dashboard，然后修改configmap的注册地址
dubbo.registry.address=zookeeper://zk2.phc-dow.com:2181
删除pod，k8s会自动生成一个新的pod，登录网页查看
可以看到注册地址根据configmap资源的修改而更改
三、交付Apollo至Kubernetes集群
1.Apollo简介
Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。
- 服务端基于Spring Boot和Spring Cloud开发，打包后可以直接运行，不需要额外安装Tomcat等应用容器。
- Java客户端不依赖任何框架，能够运行于所有Java运行时环境，同时对Spring/Spring Boot环境也有较好的支持。
- .Net客户端不依赖任何框架，能够运行于所有.Net运行时环境。
基础架构
- config service 提供配置的读取、推送等功能，服务对象是appollo客户端
- admin service提供配置的修改、发布等功能，服务对象是apollo portal管理界面
- portal是一个web界面，通过web界面配置应用程序的配置，并通过admin service写到数据库里（config db）
- portal通过portal db可以对配置支持回滚，历史版本等功能
- config service和admin service都是多实例、无状态部署，所以需要将自己注册到Eureka中并保持心跳
- 在eureka之上我们架了一层meta server用于封装eureka的服务发现接口
- client通过域名访问meta server获取config service服务列表ip+port，而后直接通过ip+port访问服务，同时在client测会做load、balance、错误重试
- portal通过域名访问meta server获取admin service服务列表ip+port，而后直接通过ip+port访问服务，同时在portal侧会做load balance、错误重试
简化模型
- apollo最重要的三个组件是portal、config service和admin service加两个数据库config db和portal db
2.配置configdb数据库
注意：MySQL版本应为5.6或以上
###安装mariadb
[root@kjdow7-11 ~]# vim /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
[root@kjdow7-11 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
[root@kjdow7-11 ~]# yum update MariaDB-server -y
###配置mariadb
[root@kjdow7-11 ~]# cat /etc/my.cnf
[mysql]
default-character-set = utf8mb4
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"
[root@kjdow7-11 ~]# systemctl start mariadb
[root@kjdow7-11 ~]# systemctl enable mariadb
[root@kjdow7-11 ~]# mysqladmin -u root password
New password: 123456
Confirm new password:123456
[root@kjdow7-11 ~]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or g.
Your MariaDB connection id is 4
Server version: 10.1.44-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
MariaDB [(none)]> s
--------------
mysql Ver 15.1 Distrib 10.1.44-MariaDB, for Linux (x86_64) using readline 5.1
Connection id: 4
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.1.44-MariaDB MariaDB Server
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 28 sec
注： 确认都是utf8mb4
###下载configdb数据库
[root@kjdow7-11 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql
###导入数据库
[root@kjdow7-11 ~]# mysql -uroot -p < V1.0.0__initialization.sql
Enter password:
###查看数据库
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [ApolloConfigDB]> show tables;
+--------------------------+
| Tables_in_ApolloConfigDB |
+--------------------------+
| App |
| AppNamespace |
| Audit |
| Cluster |
| Commit |
| GrayReleaseRule |
| Instance |
| InstanceConfig |
| Item |
| Namespace |
| NamespaceLock |
| Release |
| ReleaseHistory |
| ReleaseMessage |
| ServerConfig |
+--------------------------+
15 rows in set (0.00 sec)
####数据库用户授权
MariaDB [ApolloConfigDB]> grant select,update,insert,delete on ApolloConfigDB.* to apolloconfig@'10.4.7.%' identified by '123456';
Query OK, 0 rows affected (0.00 sec)
###修改初始数据
update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.phc-dow.com/eureka" where ServerConfig.Key="eureka.service.url";
注：默认是http://localhost:8080/eureka
授权地址是10.4.7.%是因为在k8s中flannel已经做了iptables优化
[root@kjdow7-21 ~]# iptables-save | grep POSTROUTING | grep 172.7.21.0/24
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE在这里目标地址是10.4.7.11因此需要进行源地址转换
###解析域名
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010210 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
[root@kjdow7-11 ~]# systemctl restart named
3.交付apollo-config server至k8s集群中
3.1 下载软件，并配置
###下载软件包
[root@kjdow7-200 ~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
[root@kjdow7-200 ~]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
unzip -o apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
Archive: apollo-configservice-1.5.1-github.zip
creating: /data/dockerfile/apollo-configservice/scripts/
inflating: /data/dockerfile/apollo-configservice/config/application-github.properties
inflating: /data/dockerfile/apollo-configservice/apollo-configservice.conf
inflating: /data/dockerfile/apollo-configservice/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.5.1-sources.jar
inflating: /data/dockerfile/apollo-configservice/scripts/startup.sh
inflating: /data/dockerfile/apollo-configservice/config/app.properties
inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.5.1.jar
[root@kjdow7-200 apollo-configservice]# rm -f apollo-configservice-1.5.1-sources.jar
[root@kjdow7-200 apollo-configservice]# vim config/application-github.properties
[root@kjdow7-200 apollo-configservice]# cat config/application-github.properties
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
###jdbc连接使用域名，需要解析为mysql地址
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010211 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
mysql A 10.4.7.11
##为了防止mysql更改地址要修改代码
[root@kjdow7-11 ~]# systemctl restart named
[root@kjdow7-11 ~]# dig -t A mysql.phc-dow.com @10.4.7.11 +short
10.4.7.11
3.2 更新start.sh
由于需要在k8s中部署，默认的启动脚本需要修改
[root@kjdow7-200 ~]# cd /data/dockerfile/apollo-configservice/
[root@kjdow7-200 apollo-configservice]# vi scripts/startup.sh
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ====
"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
3.3 写Dockerfile并制作镜像和推送
[root@kjdow7-200 apollo-configservice]# vim /data/dockerfile/apollo-configservice/Dockerfile
FROM harbor.phc-dow.com/public/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
[root@kjdow7-200 apollo-configservice]# docker build . -t harbor.phc-dow.com/infra/apollo-configservice:v1.5.1
[root@kjdow7-200 apollo-configservice]# docker push harbor.phc-dow.com/infra/apollo-configservice:v1.5.1
3.4 准备资源配置清单
[root@kjdow7-200 apollo-configservice]# mkdir /data/k8s-yaml/apollo-configservice
[root@kjdow7-200 apollo-configservice]# cd /data/k8s-yaml/apollo-configservice
[root@kjdow7-200 apollo-configservice]# vim cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
#eureka.serviceUrl = http://config.phc-dow.com/eureka
#apollo.eureka.server.enabled=true
#apollo.eureka.client.enabled=true
app.properties: |
appId = 100003171
[root@kjdow7-200 apollo-configservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.phc-dow.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
[root@kjdow7-200 apollo-configservice]# svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
[root@kjdow7-200 apollo-configservice]# ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
spec:
rules:
- host: config.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
3.5 应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-configservice/cm.yaml
configmap/apollo-configservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-configservice/dp.yaml
deployment.extensions/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created
3.6 验证
打开网页访问config.phc-dow.com
[root@kjdow7-21 ~]# curl http://172.7.21.7:8080/info
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}
#页面上可以看到pod的ip，curl信息如上
###数据库上查看
MariaDB [ApolloConfigDB]> show processlist;
+----+--------------+-----------------+----------------+---------+------+-------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+----+--------------+-----------------+----------------+---------+------+-------+------------------+----------+
| 17 | root | localhost | ApolloConfigDB | Query | 0 | init | show processlist | 0.000 |
| 28 | apolloconfig | 10.4.7.21:50184 | ApolloConfigDB | Sleep | 4 | | NULL | 0.000 |
| 29 | apolloconfig | 10.4.7.21:50192 | ApolloConfigDB | Sleep | 0 | | NULL | 0.000 |
| 30 | apolloconfig | 10.4.7.21:50194 | ApolloConfigDB | Sleep | 0 | | NULL | 0.000 |
| 31 | apolloconfig | 10.4.7.21:50196 | ApolloConfigDB | Sleep | 0 | | NULL | 0.000 |
| 32 | apolloconfig | 10.4.7.21:50198 | ApolloConfigDB | Sleep | 590 | | NULL | 0.000 |
| 33 | apolloconfig | 10.4.7.21:50200 | ApolloConfigDB | Sleep | 589 | | NULL | 0.000 |
| 34 | apolloconfig | 10.4.7.21:50206 | ApolloConfigDB | Sleep | 589 | | NULL | 0.000 |
| 35 | apolloconfig | 10.4.7.21:50208 | ApolloConfigDB | Sleep | 589 | | NULL | 0.000 |
| 36 | apolloconfig | 10.4.7.21:50210 | ApolloConfigDB | Sleep | 589 | | NULL | 0.000 |
| 37 | apolloconfig | 10.4.7.21:50212 | ApolloConfigDB | Sleep | 589 | | NULL | 0.000 |
+----+--------------+-----------------+----------------+---------+------+-------+------------------+----------+
11 rows in set (0.00 sec)
4、交付apollo-adminservice至k8s集群中
4.1 下载软件,并配置
[root@kjdow7-200 ~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-adminservice-1.5.1-github.zip
[root@kjdow7-200 ~]# mkdir /data/dockerfile/apollo-adminservice
[root@kjdow7-200 ~]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice
Archive: apollo-adminservice-1.5.1-github.zip
creating: /data/dockerfile/apollo-adminservice/scripts/
inflating: /data/dockerfile/apollo-adminservice/config/app.properties
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice-1.5.1-sources.jar
inflating: /data/dockerfile/apollo-adminservice/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice.conf
inflating: /data/dockerfile/apollo-adminservice/scripts/startup.sh
inflating: /data/dockerfile/apollo-adminservice/config/application-github.properties
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice-1.5.1.jar
————————————————————————————————————————————————————
[root@kjdow7-200 ~]# vi /data/dockerfile/apollo-adminservice/scripts/startup.sh
#!/bin/bash
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ====
"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
4.2 制作Docker镜像并推送
[root@kjdow7-200 scripts]# vim /data/dockerfile/apollo-adminservice/Dockerfile
FROM harbor.phc-dow.com/public/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
ADD config/ /apollo-adminservice/config
ADD scripts/ /apollo-adminservice/scripts
CMD ["/apollo-adminservice/scripts/startup.sh"]
[root@kjdow7-200 apollo-adminservice]# docker build -t harbor.phc-dow.com/infra/apollo-adminservice:v1.5.1 .
[root@kjdow7-200 apollo-adminservice]# docker push harbor.phc-dow.com/infra/apollo-adminservice:v1.5.1
4.3 创建资源配置清单
[root@kjdow7-200 apollo-adminservice]# mkdir /data/k8s-yaml/apollo-adminservice
[root@kjdow7-200 apollo-adminservice]# cd /data/k8s-yaml/apollo-adminservice
[root@kjdow7-200 apollo-adminservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
#eureka.serviceUrl = http://config.phc-dow.com/eureka
app.properties: |
appId = 100003172
[root@kjdow7-200 apollo-adminservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: infra
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-adminservice-cm
containers:
- name: apollo-adminservice
image: harbor.phc-dow.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
4.4 应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-adminservice/cm.yaml
configmap/apollo-adminservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-adminservice/dp.yaml
deployment.extensions/apollo-adminservice created
4.5 验证
打开网页config.phc-dow.com，可以看到两个服务都已经注册成功
5.交付apollo-portal至k8s集群中
5.1 配置数据库
[root@kjdow7-11 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql -O portaldb.sql
###导入数据库
MariaDB [(none)]> source ./portal.sql
MariaDB [ApolloPortalDB]> show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| ApolloPortalDB |
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
6 rows in set (0.00 sec)
MariaDB [ApolloPortalDB]> use ApolloPortalDB
Database changed
MariaDB [ApolloPortalDB]> show tables;
+--------------------------+
| Tables_in_ApolloPortalDB |
+--------------------------+
| App |
| AppNamespace |
| Authorities |
| Consumer |
| ConsumerAudit |
| ConsumerRole |
| ConsumerToken |
| Favorite |
| Permission |
| Role |
| RolePermission |
| ServerConfig |
| UserRole |
| Users |
+--------------------------+
14 rows in set (0.00 sec)
###授权用户连接数据库
MariaDB [ApolloPortalDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%" identified by "123456";
Query OK, 0 rows affected (0.00 sec)
###修改数据库
MariaDB [ApolloPortalDB]> update ApolloPortalDB.ServerConfig set value='[{"orgId":"kj-01","orgName":"前端"},{"orgId":"kj-02","orgName":"后端"},{"orgId":"kj-03","orgName":"测试"},{"orgId":"kj-04","orgName":"运维"}]' where Id=2;
Query OK, 1 row affected (0.03 sec)
Rows matched: 1 Changed: 1 Warnings: 0
5.2 下载软件，并配置
###下载软件
[root@kjdow7-200 ~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip
[root@kjdow7-200 ~]# mkdir /data/dockerfile/apollo-portal
[root@kjdow7-200 ~]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal
Archive: apollo-portal-1.5.1-github.zip
creating: /data/dockerfile/apollo-portal/scripts/
inflating: /data/dockerfile/apollo-portal/apollo-portal.conf
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.5.1.jar
inflating: /data/dockerfile/apollo-portal/scripts/startup.sh
inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties
inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-portal/config/app.properties
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.5.1-sources.jar
inflating: /data/dockerfile/apollo-portal/config/application-github.properties
[root@kjdow7-200 ~]# cd /data/dockerfile/apollo-portal
[root@kjdow7-200 apollo-portal]# vi /data/dockerfile/apollo-portal/scripts/startup.sh
#!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ====
"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
5.3 写Dockerfile并制作镜像和推送
[root@kjdow7-200 ~]# cd /data/dockerfile/apollo-portal/
[root@kjdow7-200 apollo-portal]# vi /data/dockerfile/apollo-portal/Dockerfile
FROM harbor.phc-dow.com/public/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts
CMD ["/apollo-portal/scripts/startup.sh"]
###制作镜像
[root@kjdow7-200 apollo-portal]# docker build -t harbor.phc-dow.com/infra/apollo-portal:v1.5.1 .
[root@kjdow7-200 apollo-portal]# docker push harbor.phc-dow.com/infra/apollo-portal:v1.5.1
5.4 准备资源配置清单
[root@kjdow7-200 apollo-portal]# mkdir /data/k8s-yaml/apollo-portal
[root@kjdow7-200 apollo-portal]# cd /data/k8s-yaml/apollo-portal
[root@kjdow7-200 apollo-portal]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId = 100003173
apollo-env.properties: |
dev.meta=http://config.phc-dow.com
[root@kjdow7-200 apollo-portal]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
labels:
name: apollo-portal
spec:
replicas: 1
selector:
matchLabels:
name: apollo-portal
template:
metadata:
labels:
app: apollo-portal
name: apollo-portal
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-portal-cm
containers:
- name: apollo-portal
image: harbor.phc-dow.com/infra/apollo-portal:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-portal/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
[root@kjdow7-200 apollo-portal]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-portal
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-portal
[root@kjdow7-200 apollo-portal]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
spec:
rules:
- host: portal.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: apollo-portal
servicePort: 8080
5.5 解析域名并应用资源配置清单
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010212 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
mysql A 10.4.7.11
portal A 10.4.7.10
[root@kjdow7-11 ~]# systemctl restart named
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/cm.yaml
configmap/apollo-portal-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/dp.yaml
deployment.extensions/apollo-portal created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/svc.yaml
service/apollo-portal created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/ingress.yaml
ingress.extensions/apollo-portal created
6、 操作apollo，重新集成dubbo-demo-service
6.1 在apollo上创建dubbo-demo-service项目
等待服务启动之后，打开网页portal.phc-dow.com.用户名：apollo 密码：admin123
登录后，新建项目
创建后如下图所示：
点击 “新增配置” key：dubbo.registry value: zookeeper://zk1.phc-dow.com:2181 comment "dubbo服务的注册中心地址" 选择集群 “dev”
再次点击“新增配置” key：dubbo.port value: 20880 comment "dubbo服务提供者的监听端口" 选择集群 “dev”
点击右上角的发布按钮
6.2 使用jenkins进行持续集成dubbo-demo-service
1）修改git上源码
在github上修改dubbo-demo-service的源代码，并创建名为apollo的分支
代码里引用了“dubbo.registry”和"dubbo.port"这两个变量，在apollo里的配置中心也设置了对应的值
2）登录jenkins，并重新进行部署
依次填入/选择：
app_name: dubbo-demo-service
image_name: app/dubbo-demo-service
git_repo: https://gitee.com/stanleywang/dubbo-demo-service.git
git_ver: apollo
add_tag: 202002092008
mvn_dir: ./
target_dir: ./dubbo-server/target
mvn_cmd: mvn clean package -Dmaven.test.skip=true
base_image: base/jre8:8u112
maven: 3.6.0-8u181
点击Build进行构建，等待构建完成。
jenkins集成之后，自动创建好镜像，并推送到harbor仓库成功
6.3 修改并应用新资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/dubbo-demo-service/
[root@kjdow7-200 dubbo-demo-service]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.phc-dow.com/app/dubbo-demo-service:apollo_202002092008
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.phc-dow.com
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
注：修改了两个地方，修改了image，添加了env，“name: C_OPTS”
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-service/dp.yaml
deployment.extensions/dubbo-demo-service configured
当dubbo-demo-service服务启动之后，就可以在apollo上查看到对应的实例了
登录http://dubbo-monitor.phc-dow.com进行查看
可以看到已经注册成功，并使用20880端口
6.4 在portal页面修改配置并应用
打开portal的web页面，进入dubbo-demo-service，点击修改，修改“dubbo.port”的值为20881
点击发布新的配置
在k8s集群中删除dubbo-demo-service的pod，k8s会自动拉起一个新的pod，并重新从apollo中拉取最新的配置，启动之后，重新登录http://dubbo-monitor.phc-dow.com进行查看，可以看到新的服务端口已经变为20881
7、 操作apollo，重新集成dubbo-demo-web
7.1 在apollo上创建dubbo-demo-web项目
打开网页apollo.phc-dow.com并创建项目
注意：这里的appid要跟源代码中的值完全匹配
点击添加配置
添加配置： key=dubbo.registry value=zookeeper://zk1.phc-dow.com:2181 comment=dubbo服务消费者注册中心 选择集群=DEV
7.2 使用jenkins进行持续集成dubbo-demo-web
1) 修改git源码
在github上修改dubbo-demo-web的源代码，并创建名为apollo的分支
app.id=dubbo-demo-web
dubbo.registry=$(dubbo.registry)
2）登录jenkins，并重新进行部署
依次填入/选择：
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@gitee.com:stanleywang/dubbo-demo-web.git
git_ver: apollo
add_tag: 202002101200
mvn_dir: ./
target_dir: ./dubbo-client/target
mvn_cmd: mvn clean package -Dmaven.test.skip=true
base_image: base/jre8:8u112
maven: 3.6.0-8u181
点击Build进行构建，等待构建完成。
7.3 修改并应用新资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/dubbo-demo-consumer/
[root@kjdow7-200 dubbo-demo-consumer]# vim dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.phc-dow.com/app/dubbo-demo-consumer:apollo_202002101200
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.phc-dow.com:18180
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
注：修改了两个地方，修改了image，添加了env，“name: C_OPTS”
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-consumer/dp.yaml
deployment.extensions/dubbo-demo-consumer configured
7.4 验证
等待dubbo-demo-web的新的pod启动之后，打开页面可以看到已经由一个实例使用此配置
在dubbo-monitor上可以看到已经注册到zk1上了
打开网页http://demo.phc-dow.com/hello?name=wangleiapollo应用hello应用
注：如果修改了代码，则重新操作7.2到7.3步骤
如果在portal上修改了配置，则在发布后，重启pod即可
四、 实战使用apollo分环境管理dubbo服务--交付apollo-configservice
1.环境准备
1.1 修改dubbo和apollo所有服务pod的期望值为0
修改dubbo-demo-service和dubbo-demo-web的pod期望值为0，修改apollo-portal、apollo-adminservice、apollo-configservice的pod期望值为0
1.2 zookeeper添加域名解析
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010213 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
mysql A 10.4.7.11
portal A 10.4.7.10
zk-test A 10.4.7.11
zk-prod A 10.4.7.12
[root@kjdow7-11 ~]# systemctl restart named
[root@kjdow7-11 ~]# dig -t A zk-test.phc-dow.com @10.4.7.11 +short
10.4.7.11
[root@kjdow7-11 ~]# dig -t A zk-prod.phc-dow.com @10.4.7.11 +short
10.4.7.12
添加两个域名，分作用于测试环境和生产环境
1.3 k8s集群中创建
[root@kjdow7-21 ~]# kubectl create ns test
namespace/test created
[root@kjdow7-21 ~]# kubectl create ns prod
namespace/prod created
[root@kjdow7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.phc-dow.com --docker-username=admin --docker-password=Harbor_kjdow1! -n test
secret/harbor created
[root@kjdow7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.phc-dow.com --docker-username=admin --docker-password=Harbor_kjdow1! -n prod
secret/harbor created
创建两个名称空间，分别用于测试环境和生产环境，并分别创建secret，用户从harbor上拉取镜像
2、修改数据库
2.1 配置两个环境的configdb
###修改是数据库
[root@kjdow7-11 ~]# wget -O configdb.sql https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql
[root@kjdow7-11 ~]# sed -i s#ApolloConfigDB#ApolloConfigTestDB#g configdb.sql
[root@kjdow7-11 ~]# mysql -uroot -p < configdb.sql
Enter password:
[root@kjdow7-11 ~]# sed -i s#ApolloConfigTestDB#ApolloConfigProdDB#g configdb.sql
[root@kjdow7-11 ~]# mysql -uroot -p < configdb.sql
Enter password:
###查看数据库
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| ApolloConfigProdDB |
| ApolloConfigTestDB |
| ApolloPortalDB |
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
8 rows in set (0.00 sec)
###分别给两个数据库修改连接的eureka地址
update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.phc-dow.com/eureka" where ServerConfig.Key="eureka.service.url";
update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.phc-dow.com/eureka" where ServerConfig.Key="eureka.service.url";
###分别给两个数据库授权
grant select,update,insert,delete on ApolloConfigTestDB.* to apolloconfigtest@'10.4.7.%' identified by '123456';
grant select,update,insert,delete on ApolloConfigProdDB.* to apolloconfigprod@'10.4.7.%' identified by '123456';
2.2 配置portaldb数据库
MariaDB [ApolloPortalDB]> truncate table ApolloPortalDB.AppNamespace;
Query OK, 0 rows affected (0.26 sec)
MariaDB [ApolloPortalDB]> truncate table ApolloPortalDB.App;
###这两张表中保存了之前配置的信息，先删除了
MariaDB [(none)]> update ApolloPortalDB.ServerConfig set Value='fat,pro' where Id=1;
Query OK, 1 row affected (0.05 sec)
Rows matched: 1 Changed: 1 Warnings: 0
MariaDB [(none)]> select * from ApolloPortalDB.ServerConfigG
*************************** 1. row ***************************
Id: 1
Key: apollo.portal.envs
Value: fat,pro
Comment: 可支持的环境列表
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-02-08 17:52:00
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-02-10 20:04:58
*************************** 2. row ***************************
Id: 2
Key: organizations
Value: [{"orgId":"kj-01","orgName":"前端"},{"orgId":"kj-02","orgName":"后端"},{"orgId":"kj-03","orgName":"测试"},{"orgId":"kj-04","orgName":"运维"}]
Comment: 部门列表
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-02-08 17:52:00
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-02-08 18:09:42
注：修改portaldb中的serverconfig表，让portal支持测试环境（fat）和生产环境（pro）
portaldb不支持用test字符用户环境列表
###portaldb可支持的环境列表官方提供字符如下所示
public enum Env{
LOCAL, DEV, BETA, FWS, FAT, UAT, LPT, PRO, TOOLS, UNKNOWN;
...
}
3、修改资源配置清单
3.1 修改portalservice
[root@kjdow7-200 ~]# cd /data/k8s-yaml/apollo-portal/
[root@kjdow7-200 apollo-portal]# vim cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId = 100003173
apollo-env.properties: |
fat.meta=http://config-test.phc-dow.com:18180
pro.meta=http://config-prod.phc-dow.com:18180
主要是修改最后两行
一套Portal可以管理多个环境，但是每个环境都需要独立部署一套Config Service、Admin Service和ApolloConfigDB
###应用修改后的资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/cm.yaml
configmap/apollo-portal-cm configured
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/apollo-portal/dp.yaml
deployment.extensions/apollo-portal configured
打开页面portal.phc-dow.com 可以看到之前的配置已经没有了，点击管理员工具选择“系统参数”
输入“apollo.portal.envs”可以查看到目前支持的环境是“fat.pro”
3.2 创建测试环境和生产环境的资源配置清单
1) 创建目录和域名解析
[root@kjdow7-200 ~]# cd /data/k8s-yaml/
[root@kjdow7-200 k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
mkdir: created directory ‘test’
mkdir: created directory ‘test/apollo-configservice’
mkdir: created directory ‘test/apollo-adminservice’
mkdir: created directory ‘test/dubbo-demo-service’
mkdir: created directory ‘test/dubbo-demo-consumer’
[root@kjdow7-200 k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
mkdir: created directory ‘prod’
mkdir: created directory ‘prod/apollo-configservice’
mkdir: created directory ‘prod/apollo-adminservice’
mkdir: created directory ‘prod/dubbo-demo-service’
mkdir: created directory ‘prod/dubbo-demo-consumer’
###域名解析
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010214 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
mysql A 10.4.7.11
portal A 10.4.7.10
zk-test A 10.4.7.11
zk-prod A 10.4.7.12
config-test A 10.4.7.10
config-prod A 10.4.7.10
[root@kjdow7-11 ~]# systemctl restart named
添加测试环境和开发环境连接config的域名解析
2）创建测试环境的资源配置清单并应用
A、apollo-configservice
[root@kjdow7-200 k8s-yaml]# cp /data/k8s-yaml/apollo-configservice/* /data/k8s-yaml/test/apollo-configservice/
[root@kjdow7-200 k8s-yaml]# ls /data/k8s-yaml/test/apollo-configservice/
cm.yaml dp.yaml ingress.yaml svc.yaml
[root@kjdow7-200 k8s-yaml]# cd /data/k8s-yaml/test/apollo-configservice/
[root@kjdow7-200 apollo-configservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: test
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.datasource.username = apolloconfigtest
spring.datasource.password = 123456
#apollo.eureka.server.enabled=true
#apollo.eureka.client.enabled=true
app.properties: |
appId = 100003171
##修改了namespace、spring.datasource.url、spring.datasource.username
[root@kjdow7-200 apollo-configservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: test
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.phc-dow.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace
[root@kjdow7-200 apollo-configservice]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: test
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
##修改了namespace
[root@kjdow7-200 apollo-configservice]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: test
spec:
rules:
- host: config-test.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
##修改了namespace、host
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-configservice/cm.yaml
configmap/apollo-configservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-configservice/dp.yaml
deployment.extensions/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created
B、apollo-adminservice
###配置资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/test/apollo-adminservice/
[root@kjdow7-200 apollo-adminservice]# cp /data/k8s-yaml/apollo-adminservice/* .
[root@kjdow7-200 apollo-adminservice]# ls
cm.yaml dp.yaml
[root@kjdow7-200 apollo-adminservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: test
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.datasource.username = apolloconfigtest
spring.datasource.password = 123456
#eureka.serviceUrl = http://config-test.phc-dow.com/eureka
#eureka.client.register-with-eureka=false
#eureka.client.fetch-registry=false
app.properties: |
appId = 100003172
##修改了namespace、spring.datasource.url、spring.datasource.username
[root@kjdow7-200 apollo-adminservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: test
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-adminservice-cm
containers:
- name: apollo-adminservice
image: harbor.phc-dow.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-adminservice/cm.yaml
configmap/apollo-adminservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/apollo-adminservice/dp.yaml
deployment.extensions/apollo-adminservice created
C、验证
打开页面 config-test.phc-dow.com,当以上两个服务启动之后，在页面中已经可以看到注册成功了
D、配置portal配置中心dubbo服务提供者
打开页面添加新项目appid是dubbo-demo-service (这里的appid需要跟代码中的appid完全匹配)
应用名称是dubbo服务提供者
添加好之后，可以看到这个项目有两个环境列表，点击fat配置测试环境，并点击添加配置
添加配置--> key=dubbbo.registry value=zookeeper://zk-test.phc-dow.com:2181 comment=测试环境dubbo服务提供者注册中心地址 选择 fat
添加配置--> key=dubbbo.port value=20880 comment=测试环境dubbo服务提供者监听的端口 选择 fat
提交之后 点击发布
E、配置portal配置中心dubbo服务消费者
打开页面添加新项目appid是dubbo-demo-web (这里的appid需要跟代码中的appid完全匹配)
应用名称是dubbo服务消费者
添加配置--> key=dubbbo.registry value=zookeeper://zk-test.phc-dow.com:2181 comment=测试环境dubbo服务消费者注册中心地址 选择 fat
点击发布
F、dubbo-monitor
###修改配置清单
[root@kjdow7-200 dubbo-demo-consumer]# cd /data/k8s-yaml/dubbo-monitor/
[root@kjdow7-200 dubbo-monitor]# ls
configmap.yaml dp.yaml ingress.yaml svc.yaml
[root@kjdow7-200 dubbo-monitor]# vim configmap.yaml
[root@kjdow7-200 dubbo-monitor]# pwd
/data/k8s-yaml/dubbo-monitor
[root@kjdow7-200 dubbo-monitor]# cat configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
dubbo.properties: |
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=kjdow
dubbo.registry.address=zookeeper://zk-test.phc-dow.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
dubbo.log4j.level=WARN
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/configmap.yaml
configmap/dubbo-monitor-cm configured
G、dubbo-demo-service
###配置资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/test/dubbo-demo-service/
[root@kjdow7-200 dubbo-demo-service]# cp /data/k8s-yaml/dubbo-demo-service/* .
[root@kjdow7-200 dubbo-demo-service]# ls
dp.yaml
[root@kjdow7-200 dubbo-demo-service]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: test
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.phc-dow.com/app/dubbo-demo-service:apollo_202002092008
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.phc-dow.com
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace、env-value
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/dubbo-demo-service/dp.yaml
deployment.extensions/dubbo-demo-service created
H、dubbo-demo-consumer
###配置资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/test/dubbo-demo-consumer/
[root@kjdow7-200 dubbo-demo-consumer]# cp /data/k8s-yaml/dubbo-demo-consumer/* .
[root@kjdow7-200 dubbo-demo-consumer]# ls
dp.yaml ingress.yaml svc.yaml
[root@kjdow7-200 dubbo-demo-consumer]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: test
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.phc-dow.com/app/dubbo-demo-consumer:apollo_202002101200
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.phc-dow.com:18180
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace、env-value
[root@kjdow7-200 dubbo-demo-consumer]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-demo-consumer
namespace: test
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-demo-consumer
clusterIP: None
type: ClusterIP
sessionAffinity: None
##修改了namespace
[root@kjdow7-200 dubbo-demo-consumer]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: test
spec:
rules:
- host: demo-test.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: dubbo-demo-consumer
servicePort: 8080
##修改了namespace、servicename
###配置域名解析
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010215 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
config A 10.4.7.10
mysql A 10.4.7.11
portal A 10.4.7.10
zk-test A 10.4.7.11
zk-prod A 10.4.7.12
config-test A 10.4.7.10
config-prod A 10.4.7.10
demo-test A 10.4.7.10
demo-prod A 10.4.7.10
[root@kjdow7-11 ~]# systemctl restart named
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/dubbo-demo-consumer/dp.yaml
deployment.extensions/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/dubbo-demo-consumer/svc.yaml
service/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/test/dubbo-demo-consumer/ingress.yaml
ingress.extensions/dubbo-demo-consumer created
I、验证
由于修改了dubbo-monitor的注册地址为zk-test.phc-dow.com。因此在这个页面可以看到dubbo的服务端和客户端都注册成功
打开页面demo-test.phc-dow.com/hello?name=wangleitest.在这里调用了hello的函数
3) 创建生产环境的资源配置清单并应用
A、apollo-configservice
[root@kjdow7-200 apollo-configservice]# cp ./* /data/k8s-yaml/prod/apollo-configservice/
[root@kjdow7-200 apollo-configservice]# cd /data/k8s-yaml/prod/apollo-configservice/
[root@kjdow7-200 apollo-configservice]# ls
cm.yaml dp.yaml ingress.yaml svc.yaml
[root@kjdow7-200 apollo-configservice]# vim cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: prod
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.datasource.username = apolloconfigprod
spring.datasource.password = 123456
#apollo.eureka.server.enabled=true
#apollo.eureka.client.enabled=true
app.properties: |
appId = 100003171
##修改了namespace、spring.datasource.url、spring.datasource.username
[root@kjdow7-200 apollo-configservice]# vim dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: prod
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.phc-dow.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace
[root@kjdow7-200 apollo-configservice]# vim svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: prod
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
##修改了namespace
[root@kjdow7-200 apollo-configservice]# vim ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: prod
spec:
rules:
- host: config-prod.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
##修改了namespace、host
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-configservice/cm.yaml
configmap/apollo-configservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-configservice/dp.yaml
deployment.extensions/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created
B、apollo-adminservice
###配置adminservice资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/prod/apollo-adminservice/
[root@kjdow7-200 apollo-adminservice]# cp /data/k8s-yaml/apollo-adminservice/* .
[root@kjdow7-200 apollo-adminservice]# ls
cm.yaml dp.yaml
[root@kjdow7-200 apollo-adminservice]# vim cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: prod
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.phc-dow.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.datasource.username = apolloconfigprod
spring.datasource.password = 123456
#eureka.serviceUrl = http://config-prod.phc-dow.com/eureka
#eureka.client.register-with-eureka=false
#eureka.client.fetch-registry=false
app.properties: |
appId = 100003172
##修改了namespace、spring.datasource.url、spring.datasource.username
[root@kjdow7-200 apollo-adminservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: prod
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-adminservice-cm
containers:
- name: apollo-adminservice
image: harbor.phc-dow.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-adminservice/cm.yaml
configmap/apollo-adminservice-cm created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/apollo-adminservice/dp.yaml
deployment.extensions/apollo-adminservice created
C、验证
打开页面 config-prod.phc-dow.com,当以上两个服务启动之后，在页面中已经可以看到注册成功了
D、配置portal配置中心dubbo服务提供者
dubbo-demo-service，可以看到这个项目有两个环境列表，点击pro配置测试环境，并点击添加配置
添加配置--> key=dubbbo.registry value=zookeeper://zk-prod.phc-dow.com:2181 comment=生产环境dubbo服务提供者注册中心地址 选择 pro
添加配置--> key=dubbbo.port value=20880 comment=生产环境dubbo服务提供者监听的端口 选择 pro
提交之后 点击发布
E、配置portal配置中心dubbo服务消费者
dubbo-demo-web，可以看到这个项目有两个环境列表，点击pro配置测试环境，并点击添加配置
添加配置--> key=dubbbo.registry value=zookeeper://zk-test.phc-dow.com:2181 comment=生产环境dubbo服务消费者注册中心地址 选择 pro
点击发布
F、dubbo-demo-service
###配置资源配置清单
[root@kjdow7-200 ~]# cd /data/k8s-yaml/prod/dubbo-demo-service/
[root@kjdow7-200 dubbo-demo-service]# cp /data/k8s-yaml/dubbo-demo-service/* .
[root@kjdow7-200 dubbo-demo-service]# ls
dp.yaml
[root@kjdow7-200 dubbo-demo-service]# vi dp.yaml
[root@kjdow7-200 dubbo-demo-service]# vi dp.yaml
[root@kjdow7-200 dubbo-demo-service]# cat dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: prod
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.phc-dow.com/app/dubbo-demo-service:apollo_202002092008
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=pro -Dapollo.meta=http://config-prod.phc-dow.com:18180
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/dubbo-demo-service/dp.yaml
deployment.extensions/dubbo-demo-service created
G、dubbo-demo-consumer
###配置资源配置中心
[root@kjdow7-200 ~]# cd /data/k8s-yaml/prod/dubbo-demo-consumer/
[root@kjdow7-200 dubbo-demo-consumer]# cp /data/k8s-yaml/dubbo-demo-consumer/* .
[root@kjdow7-200 dubbo-demo-consumer]# ls
dp.yaml ingress.yaml svc.yaml
[root@kjdow7-200 dubbo-demo-consumer]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: prod
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.phc-dow.com/app/dubbo-demo-consumer:apollo_202002101200
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
##修改了namespace、env-value
#__________________________________________________
注意： -Dapollo.meta=http://apollo-configservice:8080.这里没有使用域名config-prod.phc-dow.com
是因为config-prod.phc-dow.com域名经过ingress反向代理到apollo-configservice的cluster IP的8080端口，这是连接的实际地址。
在这里所有的服务都是配置在k8s集群中，刚好还在一个命名空间中，因此可以直接写短域名（完整域名是：apollo-configservice.prod.cluster.local）
#__________________________________________________
[root@kjdow7-200 dubbo-demo-consumer]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-demo-consumer
namespace: prod
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-demo-consumer
clusterIP: None
type: ClusterIP
sessionAffinity: None
##修改了namespace
[root@kjdow7-200 dubbo-demo-consumer]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: prod
spec:
rules:
- host: demo-prod.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: dubbo-demo-consumer
servicePort: 8080
##修改了namespace和host
###应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/dubbo-demo-consumer/dp.yaml
deployment.extensions/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/dubbo-demo-consumer/svc.yaml
service/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/prod/dubbo-demo-consumer/ingress.yaml
ingress.extensions/dubbo-demo-consumer created
H、验证
打开页面demo-prod.phc-dow.com/hello?name=wangleiprod.在这里调用了hello的函数
五、总结
- 配置是独立于程序的可配变量，同一份程序在不同配置下会有不同的行为
- 云原生程序的特点
-
- 程序的配置，通过设置环境变量传递到容器内部
- 程序的配置，通过程序启动参数配置生效
- 程序的配置，通过集中在配置中心进行统一管理
- Devops工程师应该做什么？
-
- 容器化公司自研的应用程序（通过docker进行二次封装）
- 推动容器化应用，转变为云原生应用（一次构建，到处使用）
- 使用容器编排框架（kubernetes），合理、规范、专业的编排业务容器