交付dubbo 微服务到k8s 集群

dubbo 微服务结构简介

上图中，蓝色的表示与业务有交互，绿色的表示只对Dubbo内部交互。上述图所描述的调用流程如下：

服务提供方发布服务到服务注册中心；
服务消费方从服务注册中心订阅服务；
服务消费方调用已经注册的可用服务

1. 部署maven 并改成国内源

在 7.200 上：
tar xf apache-maven-3.6.0-bin.tar.gz -C  /data/nfs-volume/jenkins_home/maven-3.6.3

修改maven 仓库为阿里云仓库

 <mirrors>
  
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>*</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>
  </mirrors>

#修改成阿里云镜像源

2. 准备 dubbo 底层镜像

docker pull docker.io/stanleyws/jre8:8u112  #这个是jdk8的原始镜像
 docker tag  docker.io/stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
 docker push  harbor.od.com/public/jre:8u112

cat <<EOF >/data/dockerfile/jre8/Dockerfile
FROM harbor.od.com/public/jre:8u112   
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
    echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
EOF

cat <<EOF>config.yml
---
rules:
  - pattern: '.*'
EOF 

#下载java-agent 用来给普罗米修斯监控
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar


vim entrypoint.sh
#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai-javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname-i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}

chmod +x entrypoint.sh
#登录harbor 创建base 仓库，权限是public
]# docker build . -t harbor.od.com/base/jre8:8u112 && docker push  harbor.od.com/base/jre8:8u112

3.交付dubbo server 到k8s 集群

在 jenkins 中新建一个pipline的项目，名字自定义，选择参数构建

#然后到Jenkins 新建pipline 工程，在general ，选择 this project isparameterized。

选择string parameter 。勾选 trim is string ,分别是：

app_name image_name git_repo git_ver add_tag target_dir (./target) mvn_dir (./)

mvn_cmd (mvn clean package -Dmaven.test.skip=true)

# Choice parameter

base_image (base/jre8:8u112)

maven (3.6.3) #执行编译的软件包版本,要根据实际mvn 写。

以此填入自定义的参数，我这里放个填好后的图，就是 app_name image_name 那一列：

然后在jenkins pipeline 下面填入pipeline 脚本

pipeline {
  agent any 
    stages {
      stage('pull') { //get project code from repo 
        steps {
          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
      }
      stage('build') { //exec mvn cmd
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
        }
      }
      stage('package') { //move jar file into project_dir
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
        }
      }
      stage('image') { //build image and push to registry
        steps {
          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""#这一步是把 FROM harbor.od.com/${params.base_image}
#ADD ${params.target_dir}/project_dir /opt/project_dir  具体内容写到jenkins 里面dockerfile 里面 然后在jenkins 里面构建有了新的jar 包的镜像 并推送到
#habor 上，镜像的名字后缀是刚才的add_tag 也就是时间
           sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
        }
      }
    }
}

　保存，之后再项目项目里面选择参数构建：

具体的参数如下：

　git_repo 要填dubbo-server 的git 仓库地址，git_ver 就是代码分支，这里拉取的 apolo 分支的代码。

add_tag 建议用时间日期 base_image 就是上一步构建的dockerfile 基础镜像 maven 就是之前部署的，名字要和 7.200 /data/nfs-volume/jenkins_home/ 里面 maven 的目录名字一致，要不jenkins 构建时候会找不到，类似如下报错

+ /var/jenkins_home/maven-3.6.3/bin/mvn clean package -Dmaven.test.skip=true
/var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: 1: /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: /var/jenkins_home/maven-3.6.3/bin/mvn: not found

　　然后点击构建，成功后结果如下，这时候登陆harbor 就会发现多了一个

app/dubbo-demo-service:apollo_20200517_1600 的镜像了！

4. 准备dubbo server 的yaml 文件

[root@hdss7-200 dubbo-server]# cat dp.yaml 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_20200517_1600 #就是刚才新构建的镜像
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com    #这里结合apolo 配置中心来做的
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

5 交付dubbo-monitor 到k8s 集群

#先下载dubbo monitor 包

https://github.com/Jeromefromcn/dubbo-monitor#dubbo-monitor ，下载后解压，

修改配置

cat /opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=
dubbo.registry.address=zookeeper://10.4.7.11:2181 # 改成实际zk 地址
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.log4j.file=logs/dubbo-monitor.log

　准备dockerfile

vim /data/dockerfile/dubbo-monitor/Dockerfile
FROM jeromefromcn/docker-alpine-java-bash
MAINTAINER Jerome Jiang
COPY dubbo-monitor-simple/ /dubbo-monitor-simple/   #dubbo-monitor-simple/ 就是dubbo-monitor 包里面的文件夹
CMD /dubbo-monitor-simple/bin/start.sh

docker build . -t harbor.od.com/infra/dubbo-monitor:latest #构建镜像

因为github 上 start.sh 脚本在容器环境中启动有问题需要修改，下面是修改后的。

[root@hdss7-200 dubbo-monitor]# cat dubbo-monitor-simple/bin/start.sh 
#!/bin/bash
sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties
cd `dirname $0`
BIN_DIR=`pwd`
cd ..
DEPLOY_DIR=`pwd`
CONF_DIR=$DEPLOY_DIR/conf

SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
SERVER_PROTOCOL=`sed '/dubbo.protocol.name/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
LOGS_FILE=`sed '/dubbo.log4j.file/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`

if [ -z "$SERVER_NAME" ]; then
    SERVER_NAME=`hostname`
fi

PIDS=`ps -f | grep java | grep "$CONF_DIR" |awk '{print $2}'`
if [ -n "$PIDS" ]; then
    echo "ERROR: The $SERVER_NAME already started!"
    echo "PID: $PIDS"
    exit 1
fi

if [ -n "$SERVER_PORT" ]; then
    SERVER_PORT_COUNT=`netstat -tln | grep $SERVER_PORT | wc -l`
    if [ $SERVER_PORT_COUNT -gt 0 ]; then
        echo "ERROR: The $SERVER_NAME port $SERVER_PORT already used!"
        exit 1
    fi
fi

LOGS_DIR=""
if [ -n "$LOGS_FILE" ]; then
    LOGS_DIR=`dirname $LOGS_FILE`
else
    LOGS_DIR=$DEPLOY_DIR/logs
fi
if [ ! -d $LOGS_DIR ]; then
    mkdir $LOGS_DIR
fi
STDOUT_FILE=$LOGS_DIR/stdout.log

LIB_DIR=$DEPLOY_DIR/lib
LIB_JARS=`ls $LIB_DIR|grep .jar|awk '{print "'$LIB_DIR'/"$0}'|tr "
" ":"`

JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true "
JAVA_DEBUG_OPTS=""
if [ "$1" = "debug" ]; then
    JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n "
fi
JAVA_JMX_OPTS=""
if [ "$1" = "jmx" ]; then
    JAVA_JMX_OPTS=" -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false "
fi
JAVA_MEM_OPTS=""
BITS=`java -version 2>&1 | grep -i 64-bit`
if [ -n "$BITS" ]; then
    JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=32m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
    JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi

echo -e "Starting the $SERVER_NAME ...c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1

6. 准备dobbo monitor 文件

cat deployment.yaml 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

        
[root@hdss7-200 dubbo-monitor]# cat cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties:        |
    dubbo.container=log4j,spring,registry,jetty
    dubbo.application.name=simple-monitor
    dubbo.application.owner=simplemonitor
    dubbo.registry.address=zookeeper://10.4.7.11:2181  #zk 的地址一定要和代码里面保持一致！
    dubbo.protocol.port=20880
    dubbo.jetty.port=8080
    dubbo.jetty.directory=/dubbo-monitor-simple/monitor
    dubbo.charts.directory=/dubbo-monitor-simple/charts
    dubbo.statistics.directory=/dubbo-monitor-simple/statistics
    dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
    dubbo.log4j.level=WARN
[root@hdss7-200 dubbo-monitor]# cat ingress.yaml 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  rules:
  - host: dubbo-monitor.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-monitor
          servicePort: 8080
[root@hdss7-200 dubbo-monitor]# cat svc.yaml 
kind: Service
apiVersion: v1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-monitor

添加A 记录

dubbo-monitor.od.com IN A  10.4.7.10

测试访问 
http://dubbo-monitor.od.com

7.交付 dubbo服务消费者到K8S 集群

jenkins 上新构建一个pipeline 项目，添加的参数和dubbo-serve 一样,，可以直接把dubbo-server 复制过来，然后填入参数开始构建，参数如下图：

看下 console output

Step 1/2 : FROM harbor.od.com/base/jre8:8u112
 ---> 621a01f7f85a
Step 2/2 : ADD ./dubbo-client/target/project_dir /opt/project_dir
 ---> 7d5577992ecd
Successfully built 7d5577992ecd
Successfully tagged harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650
+ docker push harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650
The push refers to repository [harbor.od.com/app/dubbo-demo-consumer]

会看到harbor 上多了  app/dubbo-demo-consumer:apollo_20200517_1650 的镜像

8. 准备consumer 的yaml 文件

# dubbo-consumer 资源配置清单

dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 #就是刚才新构建的consumer 镜像
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

cat svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer
	
cat ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 8080

kubectl apply -f 这几个文件，然后 访问 http://demo.od.com/hello?name=hello 看是否有返回结果，有返回就是正常的。

登录monitor 查看：可以看到 server 和 consumer 都有了。

相关阅读:
HDU1041
HDU1005
HDU1231
MYSQL入门总结
 oracle性能问题排查~记一个单实例的问题
 mysql案例~关于mysql的配置文件个人见解
 数据恢复系列~恢复方案制定
 mysql架构解读~mysql的多源复制
 mysql 案例~select引起的性能问题
 遭遇Bad version number in .class file
原文地址：https://www.cnblogs.com/yyselisa/p/13200844.html