apollo配置中心+dubbo改造
1.前情提要
- dubbo微服务
- 注册中心zookeeper集群
- 提供者集群
- 消费者集群
- 监控(dubbo-monitor/dubbo-admin)
- 在k8s内交付dubbo微服务的步骤
- step0：有可用的k8s集群
- step1：部署zk集群（有状态的zk，通常放在k8s集群外）
- step2：部署jenkins（以容器的形式交付在k8s集群里）root，时区，ssh-key，docker客户端，harbor连接配置
- step3：部署maven软件
- step4：制作dubbo微服务底层包
- step5：配置jenkins持续构建ci流水线
- step6：使用流水线构建羡慕，查看harbor仓库
- step7：使用资源配置清单，交付项目到k8s集群
- 交付dubbo-monitor
2.其中注意事项
- 去gitlab上拉代码可以搞一个机器人账户robot,ssh的公钥贴到deploy key,公钥贴到每个项目里,每次拉代码用到的都是这个机器人账户
- jenkins2.190.3用的docker客户端,服务端是所在宿主机的docker server,连接宿主机的docker服务端
3.陈述式方法创建configMap
kubectl create cm kubelet-cm --from-file-./kubelet.kubeconfig
4.apollo分布式配置中心
携程框架部门开源的,分布式配置中心
k8s里交付apollo就是交付一套spring cloud微服务项目
5.apollo架构
- ConfigService提供配置的读取,推送等功能,服务对象是Apoll客户端
- AdminService提供配置的修改,发布等功能,服务对象是apollo portal(管理界面)
- ConfigService和 AdminService都是多实例的,无状态部署,所以需要将自己注册到eureka并保持心跳
- 在eureka之上我们架了一层 meta server用于封装eureka的服务发现接口
- client通过域名访问meta server获取configservice服务列表(ip+port),而后直接通过ip+port访问服务,同时在client侧会做load balance,错误重试
- portal通过域名访问meta server获取admin service服务列表(ip+port),而后直接通过ip+port访问服务,同时在portal测会做load balance,错误重试
6.简化apoll模型
交付apoll到k8s集群里 ,只要交付adminService,configService,portal
7.部署依赖的mysql
7.1 安装部署mysql数据库
mysql的版本要5.6以上(交付到10.4.7.11机器上)
- 使用mariadb的yum源
vim /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = https://mirrors.aliyun.com/mariadb/yum/10.4/centos7-amd64/
gpgkey=https://mirrors.aliyun.com/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
- 导入GPG-KEY
rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
- 查看rpm包
yum list mariadb --show-duplicates
yum clean all;yum makecache
yum list mariadb-server --show-duplicates
yum install Mariadb-server -y
- 配置my.cnf
/etc/my.cnf.d/server.cnf
添加关键内容
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"
/etc/my.cnf.d/mysql-client.cnf
[mysql]
default-character-set=utf8mb4
- 启动mysql
systemctl start mariadb
systemctl enable mariadb
- 设置账户密码
mysqladmin -u root password
New password: 123456
Confirm new password:123456
mysql -u root -p123456
MariaDB[(none)]>\s
--------------
mysql Ver 15.1 Distrib 10.4.24-MariaDB, for Linux (x86_64) using readline 5.1
Connection id: 10
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.4.24-MariaDB MariaDB Server
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 1 min 20 sec
Threads: 6 Questions: 8 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 10 Queries per second avg: 0.100
--------------
MariaDB[(none)]>show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
4 rows in set (0.000 sec)
MariaDB[(none)]>drop database test;
MariaDB[(none)]>show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.001 sec)
- 查看mysql进程
ps aux | grep mysql
netstat -tlnp | grep 3306
7.2 在数据库中初始化apollo的数据库脚本
configDB的初始化脚本，1.5.1的apollo软件依赖
7.11机器上下载sql文件
wget https://raw.githubusercontent.com/apolloconfig/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O /tmp/apolloconfig.sql
# 运行sql文件
mysql -u root -p123456 < /tmp/apolloconfig.sql
mysql -u root -p123456
MariaDB[(none)]>show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.000 sec)
MariaDB[(none)]>use ApolloConfigDB;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB[(none)]>show tables;
+--------------------------+
| Tables_in_ApolloConfigDB |
+--------------------------+
| App |
| AppNamespace |
| Audit |
| Cluster |
| Commit |
| GrayReleaseRule |
| Instance |
| InstanceConfig |
| Item |
| Namespace |
| NamespaceLock |
| Release |
| ReleaseHistory |
| ReleaseMessage |
| ServerConfig |
+--------------------------+
15 rows in set (0.000 sec)
7.3 给数据库用户授权
后面应该写nodeIP网段(计算节点上iptables添加了snat规则优化)
MariaDB[(none)]>grant INSERT,DELETE,UPDATE,SELECT ON ApolloConfigDB.* to 'apolloconfig'@'%' identified by '123456';
Query OK, 0 rows affected (0.034 sec)
MariaDB[(none)]>select user,host from mysql.user;
+--------------+-------------------+
| User | Host |
+--------------+-------------------+
| apolloconfig | % |
| | jdss7-11.host.com |
| | localhost |
| mariadb.sys | localhost |
| mysql | localhost |
| root | localhost |
+--------------+-------------------+
6 rows in set (0.014 sec)
7.4 修改初始化数据
MariaDB[(none)]>use ApolloConfig;
MariaDB[(none)]>show tables;
+--------------------------+
| Tables_in_ApolloConfigDB |
+--------------------------+
| App |
| AppNamespace |
| Audit |
| Cluster |
| Commit |
| GrayReleaseRule |
| Instance |
| InstanceConfig |
| Item |
| Namespace |
| NamespaceLock |
| Release |
| ReleaseHistory |
| ReleaseMessage |
| ServerConfig |
+--------------------------+
15 rows in set (0.000 sec)
MariaDB[(none)]>select * from ServerConfig\G
*************************** 1. row ***************************
Id: 1
Key: eureka.service.url
Cluster: default
Value: http://localhost:8080/eureka/
Comment: Eureka服务Url，多个service以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:22:32
*************************** 2. row ***************************
Id: 2
Key: namespace.lock.switch
Cluster: default
Value: false
Comment: 一次发布只能有一个人修改开关
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:22:32
*************************** 3. row ***************************
Id: 3
Key: item.key.length.limit
Cluster: default
Value: 128
Comment: item key 最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:22:32
*************************** 4. row ***************************
Id: 4
Key: item.value.length.limit
Cluster: default
Value: 20000
Comment: item value最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:22:32
*************************** 5. row ***************************
Id: 5
Key: config-service.cache.enabled
Cluster: default
Value: false
Comment: ConfigService是否开启缓存，开启后能提高性能，但是会增大内存消耗！
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:22:32
5 rows in set (0.001 sec)
--- 将里面的eureka.service.url改为
MariaDB[(none)]>update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
Query OK, 1 row affected (0.029 sec)
Rows matched: 1 Changed: 1 Warnings: 0
-- 里面内容变为
*************************** 1. row ***************************
Id: 1
Key: eureka.service.url
Cluster: default
Value: http://config.od.com/eureka
Comment: Eureka服务Url，多个service以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 14:22:32
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 14:27:17
7.5 解析域名config.od.com及mysql.od.com
7.11机器上添加A记录/var/named/od.com.zone
config A 10.4.7.10
mysql A 10.4.7.11
重启named
systemctl restart named
校验域名是否生效
dig -t A config.od.com @192.168.0.2 +short
8.交付apollo-configservice
选择1.5.1版本交付一下
8.1 制作Apollo-configService的镜像
8.1.1 下载二进制包,并进行相关配置
github上找apollo的release1.5.1版本,里面assets
下载apollo-configservice-1.5.1.github.zip 二进制包到7-200机器上
cd /opt/src
wget https://objects.githubusercontent.com/github-production-release-asset-2e65be/53127403/2dd9d280-0348-11ea-8153-27170be6887f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T063314Z&X-Amz-Expires=300&X-Amz-Signature=59d74dcc35dbf925034e358f0f9a6b11afe925a5d867febf380991ea32a19df4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=53127403&response-content-disposition=attachment%3B%20filename%3Dapollo-configservice-1.5.1-github.zip&response-content-type=application%2Foctet-stream -O apollo-configservice-1.5.1.github.zip
mkdir -p /data/dockerfile/apollo-configservice
unzip -o apollo-configservice-1.5.1.github.zip -d /data/dockerfile/apollo-configservice
cd /data/dockerfile/apollo-configservice
总用量 60584
-rwxr-xr-x 1 root root 61991736 11月 9 2019 apollo-configservice-1.5.1.jar
-rwxr-xr-x 1 root root 40249 11月 9 2019 apollo-configservice-1.5.1-sources.jar
-rw-r--r-- 1 root root 57 4月 20 2017 apollo-configservice.conf
drwxr-xr-x 2 root root 65 4月 13 14:54 config
drwxr-xr-x 2 root root 43 10月 1 2019 scripts
[root@jdss7-200 apollo-configservice]# pwd
/data/dockerfile/apollo-configservice
/bin/rm apollo-configservice-1.5.1-source.zip
[root@jdss7-200 apollo-configservice]# cd config/
[root@jdss7-200 config]# ll
总用量 8
-rw-r--r-- 1 root root 289 11月 9 2019 application-github.properties
-rw-r--r-- 1 root root 30 4月 20 2017 app.properties
[root@jdss7-200 config]# cat app.properties
appId=100003171
jdkVersion=1.8
[root@jdss7-200 config]#cat application-github.properties
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
#apollo.eureka.server.enabled=true
#apollo.eureka.client.enabled=true
[root@jdss7-200 config]# cd ../scripts/
[root@jdss7-200 scripts]# ll
总用量 12
-rwxr-xr-x 1 root root 340 4月 20 2017 shutdown.sh
-rwxr-xr-x 1 root root 5316 10月 3 2019 startup.sh
[root@jdss7-200 scripts]# /bin/rm shutdown.sh
启动脚本
[root@jdss7-200 scripts]# cat startup.sh
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
SERVER_URL="http://$(hostname -i):${SERVER_PORT}"
## Adjust memory settings if necessary
export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ==== \n"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
# 文件最后添加如下内容(看似是黑洞,但是它脚本的启动过程都放到了黑洞里),可以从github的1.5.1源码包的scripts目录里k8s相关脚本复制过来
# 并且里面serviceName改为使用hostname -i命令自动获取
# -Xms128m -Xmx128m -Xss256k
# tail -f /dev/null
chmod u+x startup.sh
cd ..
8.1.2 准备dockerfile文件
Dockerfile
FROM stanleyws/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
8.1.3 打镜像,推送私有仓库
[root@jdss7-200 apollo-configservice]#docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
Sending build context to Docker daemon 62MB
Step 1/7 : FROM stanleyws/jre8:8u112
---> fa3a085d6ef1
Step 2/7 : ENV VERSION 1.5.1
---> Running in a2170d1e8cdd
Removing intermediate container a2170d1e8cdd
---> 3ca859f9be95
Step 3/7 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
---> Running in 08a9ada14954
Removing intermediate container 08a9ada14954
---> 54bc88723d8b
Step 4/7 : ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
---> a82fc808ccce
Step 5/7 : ADD config/ /apollo-configservice/config
---> 8ac19c21c7d1
Step 6/7 : ADD scripts/ /apollo-configservice/scripts
---> dcc0ba219d04
Step 7/7 : CMD ["/apollo-configservice/scripts/startup.sh"]
---> Running in d1e18bdeb757
Removing intermediate container d1e18bdeb757
---> 6c86e55d2ef1
Successfully built 6c86e55d2ef1
Successfully tagged harbor.od.com/infra/apollo-configservice:v1.5.1
[root@jdss7-200 apollo-configservice]#docker push harbor.od.com/infra/apollo-configservice:v1.5.1
The push refers to repository [harbor.od.com/infra/apollo-configservice]
3fcbd78dab82: Pushed
5aa1246c5872: Pushed
fd78e19f26e1: Pushed
60ea86e21f19: Pushed
0690f10a63a5: Pushed
c843b2cf4e12: Mounted from base/jre8
fddd8887b725: Pushed
42052a19230c: Mounted from base/jre8
8d4d1ab5ff74: Pushed
v1.5.1: digest: sha256:48fbee02930b9ddf520524c2d8524de9720ca206e0b8f17776077736abf22446 size: 2201
8.2 准备资源配置清单yaml
7-200机器上
cd /data/k8s-yaml/
mkdir -p apollo-configservice
cd apollo-configservice
说明configService提供http,adminService不提供http,portal提供http
dp.yaml(Deployment)
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/configMap
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
cm.yaml(ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od/com/eureka
app.properties: |
appId=100003171
svc.yaml(service)
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
spec:
rules:
- host: config.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
8.3应用资源配置清单
kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
通过curl 'config.od.com'可以看到eureka
9. 验证数据库中的连接
MariaDB[(none)]>show processlist;
10.交付apollo-adminservice
下载二进制包1.5.1(github.com上release是1.5.1版本)
apollo-adminservice-1.5.1-github.zip
10.1 准备源码包
7-200机器上
cd /opt/src
wget https://objects.githubusercontent.com/github-production-release-asset-2e65be/53127403/a0e34900-0348-11ea-8bd9-7ee07784035c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T072846Z&X-Amz-Expires=300&X-Amz-Signature=ea11830509d3ce57b71a20c65693a25cbc9a824c26e2af2d03a883156a78a97d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=53127403&response-content-disposition=attachment%3B%20filename%3Dapollo-adminservice-1.5.1-github.zip&response-content-type=application%2Foctet-stream -o apollo-adminservice-1.5.1-github.zip
[root@jdss7-200 src]# mkdir -pv /data/dockerfile/apollo-adminservice
mkdir: 已创建目录 "/data/dockerfile/apollo-adminservice"
[root@jdss7-200 src]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice
Archive: apollo-adminservice-1.5.1-github.zip
creating: /data/dockerfile/apollo-adminservice/scripts/
inflating: /data/dockerfile/apollo-adminservice/config/app.properties
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice-1.5.1-sources.jar
inflating: /data/dockerfile/apollo-adminservice/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice.conf
inflating: /data/dockerfile/apollo-adminservice/scripts/startup.sh
inflating: /data/dockerfile/apollo-adminservice/config/application-github.properties
inflating: /data/dockerfile/apollo-adminservice/apollo-adminservice-1.5.1.jar
[root@jdss7-200 src]# cd /data/dockerfile/apollo-adminservice
[root@jdss7-200 src]# /bin/rm apollo-adminservice-1.5.1-source.jar
[root@jdss7-200 apollo-adminservice]# cat config/application-github.properties
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
[root@jdss7-200 apollo-adminservice]# cat config/app.properties
appId=100003172
jdkVersion=1.8
启动脚本
[root@jdss7-200 apollo-adminservice]# cat scripts/startup.sh
#!/bin/bash
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://$(hostname -i):${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ==== \n"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
#修改的地方有
#SERVER_PORT=8080
#APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
# 也可以从官方github里面指定版本的scripts目录里找脚本,然后改吧改吧
10.2 Dockerfile
FROM stanleyws/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
ADD config/ /apollo-adminservice/config
ADD scripts/ /apollo-adminservice/scripts
CMD ["/apollo-adminservice/scripts/startup.sh"]
10.3 构建镜像
[root@jdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
Sending build context to Docker daemon 58.37MB
Step 1/7 : FROM stanleyws/jre8:8u112
---> fa3a085d6ef1
Step 2/7 : ENV VERSION 1.5.1
---> Using cache
---> 3ca859f9be95
Step 3/7 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
---> Using cache
---> 54bc88723d8b
Step 4/7 : ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
---> 15d33f8d949a
Step 5/7 : ADD config/ /apollo-adminservice/config
---> d7a8c9618e85
Step 6/7 : ADD scripts/ /apollo-adminservice/scripts
---> f2eb795025c3
Step 7/7 : CMD ["/apollo-adminservice/scripts/startup.sh"]
---> Running in d3f3db93af4a
Removing intermediate container d3f3db93af4a
---> 71de72a57d76
Successfully built 71de72a57d76
Successfully tagged harbor.od.com/infra/apollo-adminservice:v1.5.1
[root@jdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.5.1
The push refers to repository [harbor.od.com/infra/apollo-adminservice]
9ad84513f1fb: Pushed
41350d43d135: Pushed
8b5075b145e9: Pushed
60ea86e21f19: Mounted from infra/apollo-configservice
0690f10a63a5: Mounted from infra/apollo-configservice
c843b2cf4e12: Mounted from infra/apollo-configservice
fddd8887b725: Mounted from infra/apollo-configservice
42052a19230c: Mounted from infra/apollo-configservice
8d4d1ab5ff74: Mounted from infra/apollo-configservice
v1.5.1: digest: sha256:87e28d8e9fbd6d8c3d76dc1aa7aadaa1adf79e0fde66a5a50903a2078bfd1455 size: 2201
# 小技巧 !$ 是上一个命令的最后参数,所以也可以写成docker push !$
10.4 编写资源配置清单
7-200机器上
cd /data/k8s-yaml
mkdir -p apollo-adminservice
cd apollo-adminservice
准备ConfigMap资源配置清单cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003172
准配Deployment资源配置清单dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: infra
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice # app的lables,一般是用于和pod和service交互的时候,通过这个app的label去相互找到对方
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume # 声明卷的名字
configMap: # 卷的类型是configMap
name: apollo-adminservice-cm # configMap的名字是apollo-adminservice-cm,对应了cm.yaml文件中的name
containers:
- name: apollo-adminservice # 容器名字
image: harbor.od.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config # 挂在到哪一个挂载点
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets: # 拉私有仓库镜像需要一个secret资源,资源名字是harbor
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0 # 用root起
schedulerName: default-scheduler
strategy:
type: RollingUpdate # 滚动升级策略
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7 # 保存7个历史记录
progressDeadlineSeconds: 600 # 600秒,pod控制器不断拉pod,重试10分钟后认为你死透了
10.5 应用资源配置清单
[root@jdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/cm.yaml
configmap/apollo-adminservice-cm created
[root@jdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/dp.yaml
deployment.extensions/apollo-adminservice created
用logs看看日志
也可以用curl 'http://podIp:8080/info'来查看adminservice的状态,相当于健康检查留的口子
[root@jdss7-21 ~]# curl 'http://172.7.21.6:8080/info'
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}
11.交付apollo-portal
11.1 准备portalDB
7-11机器上的mariadb
# DataSource,这里面的db是apollo的portal的db,不是configService的db(初始化脚本在如下路径,对应了github上1.5.1版本的scripts目录里的db脚本)
# https://raw.githubusercontent.com/apolloconfig/apollo/v1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql
wget https://raw.githubusercontent.com/apolloconfig/apollo/v1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql -O /tmp/apolloportal.sql
mysql -u root -p123456 < /tmp/apolloportal.sql
mysql -u root -p123456
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| ApolloPortalDB |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.001 sec)
MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"%" identified by "123456";
Query OK, 0 rows affected (0.013 sec)
MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%" identified by "123456";
Query OK, 0 rows affected (0.004 sec)
MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"%" identified by "123456";
Query OK, 0 rows affected (0.036 sec)
MariaDB [(none)]> select user,host from mysql.user;
+--------------+-------------------+
| User | Host |
+--------------+-------------------+
| apolloconfig | % |
| apolloportal | % |
| apolloportal | 10.4.7.% |
| | jdss7-11.host.com |
| | localhost |
| mariadb.sys | localhost |
| mysql | localhost |
| root | localhost |
+--------------+-------------------+
8 rows in set (0.001 sec)
MariaDB [(none)]> use ApolloPortalDB;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [ApolloPortalDB]> update ServerConfig set Value = '[{"orgId":"od01","orgName":"linux学院"},{"orgId":"od02","orgName":"云计算学院"}]' where Id = 2;
Query OK, 1 row affected (0.018 sec)
Rows matched: 1 Changed: 1 Warnings: 0
MariaDB [ApolloPortalDB]> select * from ServerConfig\G
*************************** 1. row ***************************
Id: 1
Key: apollo.portal.envs
Value: dev
Comment: 可支持的环境列表
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
*************************** 2. row ***************************
Id: 2
Key: organizations
Value: [{"orgId":"od01","orgName":"linux学院"},{"orgId":"od02","orgName":"云计算学院"}]
Comment: 部门列表
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:13:02
*************************** 3. row ***************************
Id: 3
Key: superAdmin
Value: apollo
Comment: Portal超级管理员
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
*************************** 4. row ***************************
Id: 4
Key: api.readTimeout
Value: 10000
Comment: http接口read timeout
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
*************************** 5. row ***************************
Id: 5
Key: consumer.token.salt
Value: someSalt
Comment: consumer token salt
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
*************************** 6. row ***************************
Id: 6
Key: admin.createPrivateNamespace.switch
Value: true
Comment: 是否允许项目管理员创建私有namespace
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
*************************** 7. row ***************************
Id: 7
Key: configView.memberOnly.envs
Value: pro
Comment: 只对项目成员显示配置信息的环境列表，多个env以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2022-04-13 16:11:02
DataChange_LastModifiedBy:
DataChange_LastTime: 2022-04-13 16:11:02
7 rows in set (0.000 sec)
11.2 下载二进制包
7-200机器上
cd /opt/src
wget https://objects.githubusercontent.com/github-production-release-asset-2e65be/53127403/b3f61900-0348-11ea-89d1-b7b1e6912a75?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T081509Z&X-Amz-Expires=300&X-Amz-Signature=3014cf83fdd054f27d2909a980d701eb91a9582102b38d7092524f58193ad129&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=53127403&response-content-disposition=attachment%3B%20filename%3Dapollo-portal-1.5.1-github.zip&response-content-type=application%2Foctet-stream -o apollo-portal-1.5.1-github.zip
[root@jdss7-200 src]# mkdir -p /data/dockerfile/apollo-portal
[root@jdss7-200 src]# unzip -o ^C
[root@jdss7-200 src]# ll
总用量 174720
drwxr-xr-x 6 root root 99 4月 1 16:06 apache-maven-3.6.1
-rw-r--r-- 1 root root 9136463 4月 1 16:05 apache-maven-3.6.1-bin.tar.gz
-rw-r--r-- 1 root root 51949229 4月 13 15:29 apollo-adminservice-1.5.1-github.zip
-rw-r--r-- 1 root root 55251268 4月 13 14:33 apollo-configservice-1.5.1-github.zip
-rw-r--r-- 1 root root 39096819 4月 13 16:15 apollo-portal-1.5.1-github.zip
drwxr-xr-x 3 root root 69 7月 27 2016 dubbo-monitor
-rw-r--r-- 1 root root 23468109 4月 2 13:51 dubbo-monitor-master.zip
[root@jdss7-200 src]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal
Archive: apollo-portal-1.5.1-github.zip
creating: /data/dockerfile/apollo-portal/scripts/
inflating: /data/dockerfile/apollo-portal/apollo-portal.conf
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.5.1.jar
inflating: /data/dockerfile/apollo-portal/scripts/startup.sh
inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties
inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-portal/config/app.properties
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.5.1-sources.jar
inflating: /data/dockerfile/apollo-portal/config/application-github.properties
[root@jdss7-200 src]# cd /data/dockerfile/apollo-portal/
[root@jdss7-200 src]# /bin/rm apollo-portal-1.5.1-sources.jar
[root@jdss7-200 src]# /bin/rm apollo-portal.conf
[root@jdss7-200 src]# /bin/rm scripts/shutdown.sh
[root@jdss7-200 src]# cd config
cat app.properties
appId=100003173
application-github.properties
# DataSource,这里面的db是apollo的portal的db,不是configService的db(初始化脚本在如下路径,对应了github上1.5.1版本的scripts目录里的db脚本)
# https://raw.githubusercontent.com/apolloconfig/apollo/v1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql
spring.datasource.url=jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username=apolloportal
spring.datasource.password=123456
cat apollo-env.properties
local.meta=http://localhost:8080
dev.meta=http://fill-in-dev-meta-server:8080 # 开发环境
fat.meta=http://fill-in-fat-meta-server:8080 # 测试原件
uat.meta=http://fill-in-uat-meta-server:8080 #预发环境
lpt.meta=${lpt_meta}
pro.meta=http://fill-in-pro-meta-server:8080 # 生产环境
11.3 更新startup.sh
# 只记录差异部分
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
11.4 制作Dockerfile
7-200机器上 /data/dockerfile/apollo-portal/
FROM stanleyws/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts
CMD ["/apollo-portal/scripts/startup.sh"]
11.5 构建镜像
[root@jdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
Sending build context to Docker daemon 42.35MB
Step 1/7 : FROM stanleyws/jre8:8u112
---> fa3a085d6ef1
Step 2/7 : ENV VERSION 1.5.1
---> Using cache
---> 3ca859f9be95
Step 3/7 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
---> Running in 2b5d01745564
Removing intermediate container 2b5d01745564
---> 34a80d87070f
Step 4/7 : ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
---> c4db0b61d2f9
Step 5/7 : ADD config/ /apollo-portal/config
---> 8fa4ecd2a340
Step 6/7 : ADD scripts/ /apollo-portal/scripts
---> 1b0e8e636bb0
Step 7/7 : CMD ["/apollo-portal/scripts/startup.sh"]
---> Running in c258e5be2dc6
Removing intermediate container c258e5be2dc6
---> eca759941c4e
Successfully built eca759941c4e
Successfully tagged harbor.od.com/infra/apollo-portal:v1.5.1
[root@jdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1
The push refers to repository [harbor.od.com/infra/apollo-portal]
c86fcda67036: Pushed
63ae8134c42e: Pushed
66f11e562c96: Pushed
095cf9b3d6be: Pushed
0690f10a63a5: Mounted from infra/apollo-adminservice
c843b2cf4e12: Mounted from infra/apollo-adminservice
fddd8887b725: Mounted from infra/apollo-adminservice
42052a19230c: Mounted from infra/apollo-adminservice
8d4d1ab5ff74: Mounted from infra/apollo-adminservice
v1.5.1: digest: sha256:3846e5d2f99b56214a3558f0286acf2a1509992c1497e175a0963219d57f6a0e size: 2201
11.6 资源配置清单
7-200机器上/data/k8s-yaml/apollo-portal
cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId=100003173
apollo-env.properties: |
dev.meta=http://config.od.com
dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
labels:
name: apollo-portal
spec:
replicas: 1
selector:
matchlables:
name: apollo-portal
template:
metadata:
labels:
app: apollo-portal
name: apollo-portal
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-portal-cm
containers:
- name: apollo-portal # 容器名字
image: harbor.od.com/infra/apollo-portal:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
moutPath: /apollo-portal/config # 挂在到哪一个挂载点
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets: # 拉私有仓库镜像需要一个secret资源,资源名字是harbor
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0 # 用root起
schedulerName: default-scheduler
strategy:
type: RollingUpdate # 滚动升级策略
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7 # 保存7个历史记录
progressDeadlineSeconds: 600 # 600秒,pod控制器不断拉pod,重试10分钟后认为你死透了
ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apoll-portal
namespace: infra
spec:
rules:
- host: portal.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-portal
servicePort: 8080
11.7 解析域名portal.od.com
7-11机器上个解析域名
portal A 10.4.7.10
systemctl restart named
11.8 应用资源配置 清单
kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-portal/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
11.9 portal验证
http://portal.od.com
用户名apollo密码admin
修改账户密码,adminTools->User manage
apollo/admin123/405186254@qq.com
里面系统参数
12.用apoll进行配置的定义
12.1新建项目
12.2 需要用到apoll配置的dubbo-demo-service项目分析
dubbo-demo-service是dubbo微服务项目里的provider,服务的 提供者 ,有个apoll分支
里面2个变量dubbo.registry及dubbo.port,将来通过apoll来进行替换
12.3 新增配置,进行变量的定义
在项目里新增配置
里面2个变量(comment描述)
12.4 进行配置的发布
13.如何dubbo-demo-service的镜像里使用apoll的配置
13.1 jenkins里重新输入参数 ,进行构建
构建完后,会出来新的镜像
修改相关的资源配置 清单,(7-200机器上/data/k8s-yaml/dubbo-demo-service/dp.yaml)将dp.yaml里面image中
image: harbor.od.com/app/dubbo-demo-service:master_191201_1200
改为
image: harbor.od.com/app/dubbo-demo-service:apollo_191208_1614
同时新增一个env的环境变量
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com
# 此处新加的 这个环境变量,是为了 去连apollo的,从apollo里抓到配置并应用
# 相当于启动 java程序 的 时候 java -Denv=dev -Dapollo.meta=http://config.od.com -jar aaa.jar
13.2 使其生效
kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
13.3 生效分析
如果客户端没有配监听器的话,需要重启pod,也就是 说重启java进程才会生效
如果客户端配置了监听器的话,是可以实时生效拉取配置,并reload生效
13.4 dubbo服务的消费者客户端怎么连的 apollo的
dubbo-demo-web/dubbo-client/src/main/resources/META-INF/app.properties
app.id=dubbo-demo-web
里面的app.id要和apoll的管理界面里配置的appId要完全一致,新建了一个dubbo-demo-web的项目
里面 定义配置
重新构建dubbo-消费者 的pipeline
然后修改资源配置清单 yaml
14.apoll在实践当中的使用
让apollo的环境列表丰富起来
并且让同一个docker镜像,在不同的环境都能起来
14.1 域名解析zk-test.od.com及zk-prod.od.com,用于区分生产环境和测试环境
zk-test A 10.4.7.11
zk-prod A 10.4.7.12
14.2 将dubbo的消费者和provider的deployment都调整为scale为 0
14.3 创建2个namespace
kubectl create ns test
kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n test
kubectl create ns prod
kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n prod
用apollo,portal是可以各环境共用,adminSerivce是ConfigService是每个环境各部署一套
分 环境前,先把infra命名空间里的apollo-portal,apollo-configservice,apollo-adminservice都 停了
14.4 测试环境部署数据库,adminservice,configservice
7-11机器上
apolloconfig.sql脚本里面把数据库的名字改为ApolloConfigTestDB
mysql -u root -p123456 < apolloconfig.sql
mysql -u root -p123456
>show databases;
>use ApolloConfigTestDB;
>update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.key="eureka.service.url"
>grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
14.5 生产环境部署数据库,adminservice,configservice
apolloconfig.sql脚本里面把数据库的名字改为ApolloConfigProdDB
mysql -u root -p123456 < apolloconfig.sql
mysql -u root -p123456
>show databases;
>use ApolloConfigProdDB;
>update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.key="eureka.service.url"
>grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
14.6 portal的自己的数据库多 环境支持
mysql -u root -p123456
>use ApolloPortalDB;
>update ServerConfig sett Value='fat,pro' where Id=1;
--- fat就是测试 环境,pro是生产环境,代码里面确定了必须这么写
14.7 修改portal的yaml配置文件cm.yaml
# 将里面的核心的apollo-env.properties: |
fat.meta=http://config-test.od.com
pro.meta=http://config-prod.od.com
14.8 修改资源配置清单
7-200机器上
/data/k8s-yaml/
cd /data/k8s-yaml/
mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
cd test
cd apollo-configservice
cp /data/k8s-yaml/apollo-configservice/cm.yaml .
cp /data/k8s-yaml/apollo-configservice/dp.yaml .
cp /data/k8s-yaml/apollo-configservice/svc.yaml .
cp /data/k8s-yaml/apollo-configservice/ingress.yaml .
修改cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: test
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-test.od.com/eureka
app.properties: |
appId=100003171
修改dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: test
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
修改svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: test
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
修改ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: test
spec:
rules:
- host: config-test.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
14.9 修改域名解析
7-11机器 上
config-test A 10.4.7.10
config-prod A 10.4.7.10
systemctl restart named
14.10 测试环境把configService拉起来
kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/service.yaml
kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/ingress.yaml
14.11 生产环境yaml修改,并应用
mkdir -p /data/k8s-yaml/prod/apollo-configservice
cd /data/k8s-yaml/prod/apollo-configservice
cp ../../test/apollo-configservice/cm.yaml .
cp ../../test/apollo-configservice/dp.yaml .
cp ../../test/apollo-configservice/svc.yaml .
cp ../../test/apollo-configservice/ingress.yaml .
cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: prod
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-prod.od.com/eureka
app.properties: |
appId=100003171
dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: prod
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: prod
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: prod
spec:
rules:
- host: config-prod.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
里面修改yaml,改namespace和数据库名啥的
kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/service.yaml
kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/ingress.yaml
14.12 apollo-adminservice的测试环境和生成环境也参照上面的栗子
14.12.1 apollo-adminservice的测试环境
cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: test
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-test.od.com/eureka
app.properties: |
appId=100003172
dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: test
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice # app的lables,一般是用于和pod和service交互的时候,通过这个app的label去相互找到对方
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume # 声明卷的名字
configMap: # 卷的类型是configMap
name: apollo-adminservice-cm # configMap的名字是apollo-adminservice-cm,对应了cm.yaml文件中的name
containers:
- name: apollo-adminservice # 容器名字
image: harbor.od.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config # 挂在到哪一个挂载点
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
imagePullSecrets: # 拉私有仓库镜像需要一个secret资源,资源名字是harbor
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0 # 用root起
schedulerName: default-scheduler
strategy:
type: RollingUpdate # 滚动升级策略
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7 # 保存7个历史记录
progressDeadlineSeconds: 600 # 600秒,pod控制器不断拉pod,重试10分钟后认为你死透了
14.12.2 apollo-adminservice的生产环境
cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: prod
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-prod.od.com/eureka
app.properties: |
appId=100003172
dp.yaml
[root@jdss7-200 prod]# cat dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: prod
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice # app的lables,一般是用于和pod和service交互的时候,通过这个app的label去相互找到对方
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume # 声明卷的名字
configMap: # 卷的类型是configMap
name: apollo-adminservice-cm # configMap的名字是apollo-adminservice-cm,对应了cm.yaml文件中的name
containers:
- name: apollo-adminservice # 容器名字
image: harbor.od.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config # 挂在到哪一个挂载点
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets: # 拉私有仓库镜像需要一个secret资源,资源名字是harbor
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0 # 用root起
schedulerName: default-scheduler
strategy:
type: RollingUpdate # 滚动升级策略
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7 # 保存7个历史记录
progressDeadlineSeconds: 600 # 600秒,pod控制器不断拉pod,重试10分钟后认为你死透了
14.12.3 分别apply应用执行
kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/test/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/test/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/prod/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/prod/dp.yaml
14.12.4 效果
14.13 建议测试环境和生产 环境 放到不同的k8s环境里,而不是放2个不同的 命名空间
15.portal去连多个 环境的service
7-11数据库实例上ApolloPortalDB库
truncate table AppNamespace;
truncate table App;
portal的资源配置清单
cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId=100003173
apollo-env.properties: |
fat.meta=http://config-test.od.com
pro.meta=http://config-prod.od.com
dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
labels:
name: apollo-portal
spec:
replicas: 1
selector:
matchLabels:
name: apollo-portal
template:
metadata:
labels:
app: apollo-portal
name: apollo-portal
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-portal-cm
containers:
- name: apollo-portal # 容器名字
image: harbor.od.com/infra/apollo-portal:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-portal/config # 挂在到哪一个挂载点
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
imagePullSecrets: # 拉私有仓库镜像需要一个secret资源,资源名字是harbor
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0 # 用root起
schedulerName: default-scheduler
strategy:
type: RollingUpdate # 滚动升级策略
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7 # 保存7个历史记录
progressDeadlineSeconds: 600 # 600秒,pod控制器不断拉pod,重试10分钟后认为你死透了
svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-portal
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-portal
ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apoll-portal
namespace: infra
spec:
rules:
- host: portal.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-portal
servicePort: 8080
apply交付一下
16.测试环境交付dubbo-demo-consumer和 dubbo-demo-provider
16.1 dubbo-demo-provider
dubbo-demo-provider的测试环境dp.yaml里面C_OPTS的环境变量
name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.od.com
infra命名空间 里的,configMap里面有一个dubbo-monitor的configMap,里面切换zk的节点地址就能看到dubbo-monitor连的具体是测试环境还是生产环境
16.2 dubbo-demo-consumer测试环境
dp.yaml
name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.od.com
svc.yaml
namespace: test
就可以通过 demo-test.od.com/hello?name=test来 验证过
17.生产环境交付 dubbo-demo-provider,dubbo-demo-consumer
参考测试环境的部署范例,域名都是-prod
18.提测,发版流程
测试环境test的namespace,进行jenkins构建,出来镜像后 ,镜像地址替换测试环境的 yaml文件,然后测试环境测试
测试通过了 ,线上环境直接 替换镜像地址就好了,不用再重新jenkins构建了