• （转）实验文档3：在kubernetes集群里集成Apollo配置中心

    使用ConfigMap管理应用配置

    拆分环境

    主机名角色ip
    HDSS7-11.host.com zk1.od.com(Test环境) 10.4.7.11
    HDSS7-12.host.com zk2.od.com(Prod环境) 10.4.7.12

    重配zookeeper

    HDSS7-11.host.com上：

    /opt/zookeeper/conf/zoo.cfg 
    1
    2
    3
    4
    5
    6
    		 
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper/data
    dataLogDir=/data/zookeeper/logs
    clientPort=2181

    HDSS7-12.host.com上：

    /opt/zookeeper/conf/zoo.cfg 
    1
    2
    3
    4
    5
    6
    		 
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper/data
    dataLogDir=/data/zookeeper/logs
    clientPort=2181

    重启zk(删除数据文件)

    1
    2
    3
    		 
    [root@hdss7-11 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
    [root@hdss7-12 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
    [root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop

    准备资源配置清单(dubbo-monitor)

    在运维主机HDSS7-200.host.com上：

    vi /data/k8s-yaml/dubbo-monitor/configmap.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    		 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: dubbo-monitor-cm
      namespace: infra
    data:
      dubbo.properties: |
        dubbo.container=log4j,spring,registry,jetty
        dubbo.application.name=simple-monitor
        dubbo.application.owner=
        dubbo.registry.address=zookeeper://zk1.od.com:2181
        dubbo.protocol.port=20880
        dubbo.jetty.port=8080
        dubbo.jetty.directory=/dubbo-monitor-simple/monitor
        dubbo.charts.directory=/dubbo-monitor-simple/charts
        dubbo.statistics.directory=/dubbo-monitor-simple/statistics
        dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
        dubbo.log4j.level=WARN

    vi /data/k8s-yaml/dubbo-monitor/deployment.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: dubbo-monitor
      namespace: infra
      labels: 
        name: dubbo-monitor
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: dubbo-monitor
      template:
        metadata:
          labels: 
            app: dubbo-monitor
            name: dubbo-monitor
        spec:
          containers:
          - name: dubbo-monitor
            image: harbor.od.com/infra/dubbo-monitor:latest
            ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 20880
              protocol: TCP
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - name: configmap-volume
                mountPath: /dubbo-monitor-simple/conf
          volumes:
            - name: configmap-volume
              configMap:
                name: dubbo-monitor-cm
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    应用资源配置清单

    在任意一台k8s运算节点执行：

    1
    2
    3
    4
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/configmap.yaml
    configmap/dubbo-monitor-cm created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/deployment.yaml
    deployment.extensions/dubbo-monitor configured

    重新发版，修改dubbo项目的配置文件

    修改项目源代码

    • duboo-demo-service

      dubbo-server/src/main/java/config.properties 
      1
      2
      		 
      dubbo.registry=zookeeper://zk1.od.com:2181
      dubbo.port=28080

    • dubbo-demo-web

      dubbo-client/src/main/java/config.properties 
      1
      		 
      dubbo.registry=zookeeper://zk1.od.com:2181

    使用Jenkins进行CI

    略

    修改/应用资源配置清单

    k8s的dashboard上，修改deployment使用的容器版本，提交应用

    验证configmap的配置

    在K8S的dashboard上，修改dubbo-monitor的configmap配置为不同的zk，重启POD，浏览器打开http://dubbo-monitor.od.com 观察效果

    交付Apollo至Kubernetes集群

    Apollo简介

    Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。

    官方GitHub地址

    Apollo官方地址
    官方release包

    基础架构

    apollo基础架构

    简化模型

    apollo简化架构

    交付apollo-configservice

    准备软件包

    在运维主机HDSS7-200.host.com上：
    下载官方release包

    /opt/src 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    		 
    [root@hdss7-200 src]# ls -l|grep apollo
    -rw-r--r-- 1 root root 52713404 Feb 16 23:29 apollo-configservice-1.3.0-github.zip
    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice && unzip -o apollo-configservice-1.3.0-github.zip -d /data/dockerfile/apollo-configservice
    Archive:  apollo-configservice-1.3.0-github.zip
       creating: /data/dockerfile/apollo-configservice/scripts/
      inflating: /data/dockerfile/apollo-configservice/config/application-github.properties  
      inflating: /data/dockerfile/apollo-configservice/scripts/shutdown.sh  
      inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0-sources.jar  
      inflating: /data/dockerfile/apollo-configservice/scripts/startup.sh  
      inflating: /data/dockerfile/apollo-configservice/config/app.properties  
      inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0.jar  
      inflating: /data/dockerfile/apollo-configservice/apollo-configservice.conf

    执行数据库脚本

    在数据库主机HDSS7-11.host.com上：
    注意：MySQL版本应为5.6或以上！

    • 更新yum源
    /etc/yum.repos.d/MariaDB.repo 
    1
    2
    3
    4
    5
    		 
    [mariadb]
    name = MariaDB
    baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
    gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
    gpgcheck=1

    • 导入GPG-KEY

      1
      		 
      [root@hdss7-11 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB

    • 更新数据库版本

      1
      		 
      [root@hdss7-11 ~]# yum update MariaDB-server -y

    数据库脚本地址

    1
    2
    3
    		 
    [root@hdss7-11 ~]# mysql -uroot -p
    mysql> create database ApolloConfigDB;
    mysql> source ./apolloconfig.sql

    数据库用户授权

    1
    		 
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

    修改初始数据

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    		 
    mysql> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    Query OK, 1 row affected (0.00 sec)
    Rows matched: 1  Changed: 1  Warnings: 0

    mysql> select * from ServerConfigG
    *************************** 1. row ***************************
                           Id: 1
                          Key: eureka.service.url
                      Cluster: default
                        Value: http://config.od.com/eureka
                      Comment: Eureka服务Url，多个service以英文逗号分隔
                    IsDeleted:  
         DataChange_CreatedBy: default
       DataChange_CreatedTime: 2019-04-10 15:07:34
    DataChange_LastModifiedBy: 
          DataChange_LastTime: 2019-04-11 16:28:57

    制作Docker镜像

    在运维主机HDSS7-200.host.com上：

    • 配置数据库连接串

      /data/dockerfile/apollo-configservice 
      1
      		 
      [root@hdss7-200 apollo-configservice]# cat config/application-github.properties

    • 更新startup.sh

      /data/dockerfile/apollo-configservice/scripts/startup.sh 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      		 
      #!/bin/bash
      SERVICE_NAME=apollo-configservice
      ## Adjust log dir if necessary
      LOG_DIR=/opt/logs/apollo-config-server
      ## Adjust server port if necessary
      SERVER_PORT=8080
      APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
      SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

      ## Adjust memory settings if necessary
      #export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

      ## Only uncomment the following when you are using server jvm
      #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

      ########### The following is the same for configservice, adminservice, portal ###########
      export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
      export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

      # Find Java
      if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
          javaexe="$JAVA_HOME/bin/java"
      elif type -p java > /dev/null 2>&1; then
          javaexe=$(type -p java)
      elif [[ -x "/usr/bin/java" ]];  then
          javaexe="/usr/bin/java"
      else
          echo "Unable to find Java"
          exit 1
      fi

      if [[ "$javaexe" ]]; then
          version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
          version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
          # now version is of format 009003 (9.3.x)
          if [ $version -ge 011000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 010000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 009000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          else
              JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
              JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
              JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
          fi
      fi

      printf "$(date) ==== Starting ==== 
"

      cd `dirname $0`/..
      chmod 755 $SERVICE_NAME".jar"
      ./$SERVICE_NAME".jar" start

      rc=$?;

      if [[ $rc != 0 ]];
      then
          echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
          exit $rc;
      fi

      tail -f /dev/null

    • 写Dockerfile

      /data/dockerfile/apollo-configservice/Dockerfile 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      		 
      FROM stanleyws/jre8:8u112

      ENV VERSION 1.3.0

      RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
          echo "Asia/Shanghai" > /etc/timezone

      ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
      ADD config/ /apollo-configservice/config
      ADD scripts/ /apollo-configservice/scripts

      CMD ["/apollo-configservice/scripts/startup.sh"]

    • 制作镜像并推送

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      		 
      [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.3.0
      Sending build context to Docker daemon 61.91 MB
      Step 1 : FROM stanleyws/jre8:8u112
       ---> fa3a085d6ef1
      Step 2 : ENV VERSION 1.3.0
       ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
       ---> Running in 685d51b5adb4
       ---> feb4c0289f04
      Removing intermediate container 685d51b5adb4
      Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
       ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
       ---> Running in eaa05073feeb
       ---> a3e3fd61ae35
      Removing intermediate container eaa05073feeb
      Step 4 : ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
       ---> be09a59b83a2
      Removing intermediate container ac6b8af3979b
      Step 5 : ADD config/ /apollo-configservice/config
       ---> fb64fc0f3194
      Removing intermediate container b73c5315ad20
      Step 6 : ADD scripts/ /apollo-configservice/scripts
       ---> 96ff3d9b9456
      Removing intermediate container 67ba203b3101
      Step 7 : CMD /apollo-configservice/scripts/startup.sh
       ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
       ---> Running in 80bd3f53fefc
       ---> 551ea2ba8de3
      Removing intermediate container 80bd3f53fefc
      Successfully built 551ea2ba8de3

      [root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.3.0
      The push refers to a repository [harbor.od.com/infra/apollo-configservice]
      25efb9a44683: Pushed 
      b3572bb46247: Pushed 
      e7994b936025: Pushed 
      0ff1d078cbc4: Pushed 
      ebfb473df5c2: Pushed 
      aae5c057d1b6: Pushed 
      dee6aef5c2b6: Pushed 
      a464c54f93a9: Pushed 
      v1.3.0: digest: sha256:6a8e4fdda58de0dfba9985ebbf91c4d6f46f5274983d2efa8853b03f4e45fa06 size: 1992

    解析域名

    DNS主机HDSS7-11.host.com上：

    /var/named/od.com.zone 
    1
    2
    		 
    mysql   60 IN A 10.4.7.11
    config	60 IN A 10.4.7.10

    准备资源配置清单

    在运维主机HDSS7-200.host.com上

    /data/k8s-yaml 
    1
    		 
    [root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-configservice && cd /data/k8s-yaml/apollo-configservice

    vi deployment.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-configservice
      namespace: infra
      labels: 
        name: apollo-configservice
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: apollo-configservice
      template:
        metadata:
          labels: 
            app: apollo-configservice 
            name: apollo-configservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-configservice-cm
          containers:
          - name: apollo-configservice
            image: harbor.od.com/infra/apollo-configservice:v1.3.0
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-configservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    vi svc.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    		 
    kind: Service
    apiVersion: v1
    metadata: 
      name: apollo-configservice
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector: 
        app: apollo-configservice
      clusterIP: None
      type: ClusterIP
      sessionAffinity: None

    vi ingress.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    		 
    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata: 
      name: apollo-configservice
      namespace: infra
    spec:
      rules:
      - host: config.od.com
        http:
          paths:
          - path: /
            backend: 
              serviceName: apollo-configservice
              servicePort: 8080

    vi configmap.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    		 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-configservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config.od.com/eureka
      app.properties: |
        appId=100003171

    应用资源配置清单

    在任意一台k8s运算节点执行：

    1
    2
    3
    4
    5
    6
    7
    8
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/configmap.yaml
    configmap/apollo-configservice-cm created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/deployment.yaml
    deployment.extensions/apollo-configservice created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
    service/apollo-configservice created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
    ingress.extensions/apollo-configservice created

    浏览器访问

    http://config.od.com

    交付apollo-adminservice

    准备软件包

    在运维主机HDSS7-200.host.com上：
    下载官方release包

    1
    2
    3
    4
    5
    		 
    [root@hdss7-200 src]# ls -l|grep apollo
    -rw-r--r-- 1 root root 52713404 Feb 16 08:47 apollo-configservice-1.3.0-github.zip
    -rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice && unzip -o apollo-adminservice-1.3.0-github.zip -d /data/dockerfile/apollo-adminservice

    制作Docker镜像

    在运维主机HDSS7-200.host.com上：

    • 配置数据库连接串

      /data/dockerfile/apollo-adminservice 
      1
      		 
      [root@hdss7-200 apollo-adminservice]# cat config/application-github.properties

    • 更新starup.sh

      /data/dockerfile/apollo-adminservice/scripts/startup.sh 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      		 
      #!/bin/bash
      SERVICE_NAME=apollo-adminservice
      ## Adjust log dir if necessary
      LOG_DIR=/opt/logs/apollo-adminservice
      ## Adjust server port if necessary
      SERVER_PORT=8080
      APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
      # SERVER_URL="http://localhost:${SERVER_PORT}"
      SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

      ## Adjust memory settings if necessary
      #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

      ## Only uncomment the following when you are using server jvm
      #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

      ########### The following is the same for configservice, adminservice, portal ###########
      export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
      export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

      # Find Java
      if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
          javaexe="$JAVA_HOME/bin/java"
      elif type -p java > /dev/null 2>&1; then
          javaexe=$(type -p java)
      elif [[ -x "/usr/bin/java" ]];  then
          javaexe="/usr/bin/java"
      else
          echo "Unable to find Java"
          exit 1
      fi

      if [[ "$javaexe" ]]; then
          version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
          version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
          # now version is of format 009003 (9.3.x)
          if [ $version -ge 011000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 010000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 009000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          else
              JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
              JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
              JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
          fi
      fi

      printf "$(date) ==== Starting ==== 
"

      cd `dirname $0`/..
      chmod 755 $SERVICE_NAME".jar"
      ./$SERVICE_NAME".jar" start

      rc=$?;

      if [[ $rc != 0 ]];
      then
          echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
          exit $rc;
      fi

      tail -f /dev/null

    • 写Dockerfile

      /data/dockerfile/apollo-adminservice/Dockerfile 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      		 
      FROM stanleyws/jre8:8u112

      ENV VERSION 1.3.0

      RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
          echo "Asia/Shanghai" > /etc/timezone

      ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
      ADD config/ /apollo-adminservice/config
      ADD scripts/ /apollo-adminservice/scripts

      CMD ["/apollo-adminservice/scripts/startup.sh"]

    • 制作镜像并推送

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      		 
      [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.3.0
      Sending build context to Docker daemon 58.31 MB
      Step 1 : FROM stanleyws/jre8:8u112
       ---> fa3a085d6ef1
      Step 2 : ENV VERSION 1.3.0
       ---> Using cache
       ---> feb4c0289f04
      Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
       ---> Using cache
       ---> a3e3fd61ae35
      Step 4 : ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
       ---> 6a1eb9565777
      Removing intermediate container 7196df9af6af
      Step 5 : ADD config/ /apollo-adminservice/config
       ---> 9f364b732d46
      Removing intermediate container 9b24669c6c78
      Step 6 : ADD scripts/ /apollo-adminservice/scripts
       ---> b7bc5517b0fc
      Removing intermediate container f3e34e759148
      Step 7 : CMD /apollo-adminservice/scripts/startup.sh
       ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
       ---> Running in 18c6597914b4
       ---> 82145db3ee88
      Removing intermediate container 18c6597914b4
      Successfully built 82145db3ee88

      [root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
      docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
      The push refers to a repository [harbor.od.com/infra/apollo-adminservice]
      19b1ca6c066d: Pushed 
      8fa6cde49908: Pushed 
      0b2c9b9226cc: Pushed 
      ebfb473df5c2: Mounted from infra/apollo-configservice 
      aae5c057d1b6: Mounted from infra/apollo-configservice 
      dee6aef5c2b6: Mounted from infra/apollo-configservice 
      a464c54f93a9: Mounted from infra/apollo-configservice 
      v1.3.0: digest: sha256:75367caab9bad3d0d281eb3324451a0734e84b6aa3ee860e38ad758d7166a7d1 size: 1785

    准备资源配置清单

    在运维主机HDSS7-200.host.com上

    /data/k8s-yaml 
    1
    		 
    [root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-adminservice && cd /data/k8s-yaml/apollo-adminservice

    vi deployment.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-adminservice
      namespace: infra
      labels: 
        name: apollo-adminservice
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: apollo-adminservice
      template:
        metadata:
          labels: 
            app: apollo-adminservice 
            name: apollo-adminservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-adminservice-cm
          containers:
          - name: apollo-adminservice
            image: harbor.od.com/infra/apollo-adminservice:v1.3.0
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-adminservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    vi configmap.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    		 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-adminservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config.od.com/eureka
      app.properties: |
        appId=100003172

    应用资源配置清单

    在任意一台k8s运算节点执行：

    1
    2
    3
    4
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/configmap.yaml
    configmap/apollo-adminservice-cm created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/deployment.yaml
    deployment.extensions/apollo-adminservice created

    浏览器访问

    http://config.od.com
    apollo注册中心

    交付apollo-portal

    准备软件包

    在运维主机HDSS7-200.host.com上：
    下载官方release包

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    		 
    [root@hdss7-200 src]# ls -l|grep apollo
    -rw-r--r-- 1 root root 52713404 Feb 16 08:37 apollo-configservice-1.3.0-github.zip
    -rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip
    -rw-r--r-- 1 root root 36459359 Feb 16 10:00 apollo-portal-1.3.0-github.zip

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal && unzip -o apollo-portal-1.3.0-github.zip -d /data/dockerfile/apollo-portal
    Archive:  apollo-portal-1.3.0-github.zip
      inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh  
      inflating: /data/dockerfile/apollo-portal/apollo-portal.conf  
      inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0-sources.jar  
       creating: /data/dockerfile/apollo-portal/config/
      inflating: /data/dockerfile/apollo-portal/config/application-github.properties  
      inflating: /data/dockerfile/apollo-portal/scripts/startup.sh  
      inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties  
      inflating: /data/dockerfile/apollo-portal/config/app.properties  
      inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0.jar

    执行数据库脚本

    在数据库主机HDSS7-11.host.com上：
    数据库脚本地址

    1
    2
    3
    		 
    [root@hdss7-11 ~]# mysql -uroot -p
    mysql> create database ApolloPortalDB;
    mysql> source ./apolloportal.sql

    数据库用户授权

    1
    		 
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"172.7.%" identified by "123456";

    制作Docker镜像

    在运维主机HDSS7-200.host.com上：

    • 配置数据库连接串

      /data/dockerfile/apollo-portal 
      1
      		 
      [root@hdss7-200 apollo-portal]# cat config/application-github.properties

    • 配置Portal的meta service

      /data/dockerfile/apollo-portal/config/apollo-env.properties 
      1
      		 
      dev.meta=http://config.od.com

    • 更新starup.sh

      /data/dockerfile/apollo-portal/scripts/startup.sh 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      		 
      #!/bin/bash
      SERVICE_NAME=apollo-portal
      ## Adjust log dir if necessary
      LOG_DIR=/opt/logs/apollo-portal-server
      ## Adjust server port if necessary
      SERVER_PORT=8080
      APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
      # SERVER_URL="http://localhost:$SERVER_PORT"
      SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

      ## Adjust memory settings if necessary
      #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

      ## Only uncomment the following when you are using server jvm
      #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

      ########### The following is the same for configservice, adminservice, portal ###########
      export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
      export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

      # Find Java
      if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
          javaexe="$JAVA_HOME/bin/java"
      elif type -p java > /dev/null 2>&1; then
          javaexe=$(type -p java)
      elif [[ -x "/usr/bin/java" ]];  then
          javaexe="/usr/bin/java"
      else
          echo "Unable to find Java"
          exit 1
      fi

      if [[ "$javaexe" ]]; then
          version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
          version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
          # now version is of format 009003 (9.3.x)
          if [ $version -ge 011000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 010000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          elif [ $version -ge 009000 ]; then
              JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
          else
              JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
              JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
              JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
          fi
      fi

      printf "$(date) ==== Starting ==== 
"

      cd `dirname $0`/..
      chmod 755 $SERVICE_NAME".jar"
      ./$SERVICE_NAME".jar" start

      rc=$?;

      if [[ $rc != 0 ]];
      then
          echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
          exit $rc;
      fi

      tail -f /dev/null

    • 写Dockerfile

    /data/dockerfile/apollo-portal/Dockerfile 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    		 
    FROM stanleyws/jre8:8u112

    ENV VERSION 1.3.0

    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
        echo "Asia/Shanghai" > /etc/timezone

    ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
    ADD config/ /apollo-portal/config
    ADD scripts/ /apollo-portal/scripts

    CMD ["/apollo-portal/scripts/startup.sh"]
    • 制作镜像并推送
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    		 
    [root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.3.0
    Sending build context to Docker daemon 43.35 MB
    Step 1 : FROM stanleyws/jre8:8u112
     ---> fa3a085d6ef1
    Step 2 : ENV VERSION 1.3.0
     ---> Using cache
     ---> feb4c0289f04
    Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
     ---> Using cache
     ---> a3e3fd61ae35
    Step 4 : ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
     ---> cfcf63e8eedc
    Removing intermediate container 860b55bd3fc5
    Step 5 : ADD config/ /apollo-portal/config
     ---> 3ee780369431
    Removing intermediate container 6b67ee4224b5
    Step 6 : ADD scripts/ /apollo-portal/scripts
     ---> 42c9aea2e9e3
    Removing intermediate container 2dcf8d1bf4cf
    Step 7 : CMD /apollo-portal/scripts/startup.sh
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in 9162dab8b63a
     ---> 0c020b79c36f
    Removing intermediate container 9162dab8b63a
    Successfully built 0c020b79c36f
    [root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.3.0
    docker push harbor.od.com/infra/apollo-portal:v1.3.0
    The push refers to a repository [harbor.od.com/infra/apollo-portal]
    e7c0e96ded4e: Pushed 
    0076c5344476: Pushed 
    3851a45d7440: Pushed 
    ebfb473df5c2: Mounted from infra/apollo-adminservice 
    aae5c057d1b6: Mounted from infra/apollo-adminservice 
    dee6aef5c2b6: Mounted from infra/apollo-adminservice 
    a464c54f93a9: Mounted from infra/apollo-adminservice 
    v1.3.0: digest: sha256:1aa30aac8642cceb97c053b7d74632240af08f64c49b65d8729021fef65628a4 size: 1785

    解析域名

    DNS主机HDSS7-11.host.com上：

    /var/named/od.com.zone 
    1
    		 
    portal	60 IN A 10.4.7.10

    准备资源配置清单

    在运维主机HDSS7-200.host.com上

    /data/k8s-yaml 
    1
    		 
    [root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-portal && cd /data/k8s-yaml/apollo-portal

    vi deployment.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-portal
      namespace: infra
      labels: 
        name: apollo-portal
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: apollo-portal
      template:
        metadata:
          labels: 
            app: apollo-portal 
            name: apollo-portal
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-portal-cm
          containers:
          - name: apollo-portal
            image: harbor.od.com/infra/apollo-portal:v1.3.0
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-portal/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    vi svc.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    		 
    kind: Service
    apiVersion: v1
    metadata: 
      name: apollo-portal
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector: 
        app: apollo-portal
      clusterIP: None
      type: ClusterIP
      sessionAffinity: None

    vi ingress.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    		 
    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata: 
      name: apollo-portal
      namespace: infra
    spec:
      rules:
      - host: portal.od.com
        http:
          paths:
          - path: /
            backend: 
              serviceName: apollo-portal
              servicePort: 8080

    vi configmap.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    		 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-portal-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
        spring.datasource.username = apolloportal
        spring.datasource.password = 123456
      app.properties: |
        appId=100003173
      apollo-env.properties: |
        dev.meta=http://config.od.com

    应用资源配置清单

    在任意一台k8s运算节点执行：

    1
    2
    3
    4
    5
    6
    7
    8
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/configmap.yaml
    configmap/apollo-portal-cm created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/deployment.yaml
    deployment.extensions/apollo-portal created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
    service/apollo-portal created
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
    ingress.extensions/apollo-portal created

    浏览器访问

    http://portal.od.com

    • 用户名：apollo
    • 密码： admin

    apollo-portal

    实战dubbo微服务接入Apollo配置中心

    改造dubbo-demo-service项目

    使用IDE拉取项目（这里使用git bash作为范例）

    1
    		 
    $ git clone git@gitee.com/stanleywang/dubbo-demo-service.git

    切到apollo分支

    1
    		 
    $ git checkout -b apollo

    修改pom.xml

    • 加入apollo客户端jar包的依赖
    dubbo-server/pom.xml 
    1
    2
    3
    4
    5
    		 
    <dependency>
      <groupId>com.ctrip.framework.apollo</groupId>
      <artifactId>apollo-client</artifactId>
      <version>1.1.0</version>
    </dependency>
    • 修改resource段
    dubbo-server/pom.xml 
    1
    2
    3
    4
    5
    6
    7
    		 
    <resource>
      <directory>src/main/resources</directory>
      <includes>
      <include>**/*</include>
      </includes>
      <filtering>false</filtering>
    </resource>

    增加resources目录

    /d/workspace/dubbo-demo-service/dubbo-server/src/main 
    1
    2
    3
    		 
    $ mkdir -pv resources/META-INF
    mkdir: created directory 'resources'
    mkdir: created directory 'resources/META-INF'

    修改config.properties文件

    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/config.properties 
    1
    2
    		 
    dubbo.registry=${dubbo.registry}
    dubbo.port=${dubbo.port}

    修改srping-config.xml文件

    • beans段新增属性
    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml 
    1
    		 
    xmlns:apollo="http://www.ctrip.com/schema/apollo"
    • xsi:schemaLocation段内新增属性
    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml 
    1
    		 
    http://www.ctrip.com/schema/apollo http://www.ctrip.com/schema/apollo.xsd
    • 新增配置项
    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml 
    1
    		 
    <apollo:config/>
    • 删除配置项（注释）
    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml 
    1
    		 
    <!-- <context:property-placeholder location="classpath:config.properties"/> -->

    增加app.properties文件

    /d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/META-INF/app.properties 
    1
    		 
    app.id=dubbo-demo-service

    提交git中心仓库（gitee）

    1
    		 
    $ git push origin apollo

    配置apollo-portal

    创建项目

    • 部门

      样例部门1（TEST1）

    • 应用id

      dubbo-demo-service

    • 应用名称

      dubbo服务提供者

    • 应用负责人

      apollo|apollo

    • 项目管理员

      apollo|apollo

    提交

    进入配置页面

    新增配置项1

    • Key

      dubbo.registry

    • Value

      zookeeper://zk1.od.com:2181

    • 选择集群

      DEV

    提交

    新增配置项2

    • Key

      dubbo.port

    • Value

      20880

    • 选择集群

      DEV

    提交

    发布配置

    点击发布，配置生效
    apollo-release

    使用jenkins进行CI

    略（注意记录镜像的tag）

    上线新构建的项目

    准备资源配置清单

    运维主机HDSS7-200.host.com上：

    /data/k8s-yaml/dubbo-demo-service/deployment.yaml 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: dubbo-demo-service
      namespace: app
      labels: 
        name: dubbo-demo-service
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: dubbo-demo-service
      template:
        metadata:
          labels: 
            app: dubbo-demo-service
            name: dubbo-demo-service
        spec:
          containers:
          - name: dubbo-demo-service
            image: harbor.od.com/app/dubbo-demo-service:apollo_190119_1815
            ports:
            - containerPort: 20880
              protocol: TCP
            env:
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config.od.com
            - name: JAR_BALL
              value: dubbo-server.jar
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    注意：增加了env段配置
    注意：docker镜像新版的tag

    应用资源配置清单

    在任意一台k8s运算节点上执行：

    1
    2
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/deployment.yaml
    deployment.extensions/dubbo-demo-service configured

    观察项目运行情况

    http://dubbo-monitor.od.com

    改造dubbo-demo-web

    略

    配置apollo-portal

    创建项目

    • 部门

      样例部门1（TEST1）

    • 应用id

      dubbo-demo-web

    • 应用名称

      dubbo服务消费者

    • 应用负责人

      apollo|apollo

    • 项目管理员

      apollo|apollo

    提交

    进入配置页面

    新增配置项1

    • Key

      dubbo.registry

    • Value

      zookeeper://zk1.od.com:2181

    • 选择集群

      DEV

    提交

    发布配置

    点击发布，配置生效

    使用jenkins进行CI

    略（注意记录镜像的tag）

    上线新构建的项目

    准备资源配置清单

    运维主机HDSS7-200.host.com上：

    /data/k8s-yaml/dubbo-demo-consumer/deployment.yaml 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    		 
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: dubbo-demo-consumer
      namespace: app
      labels: 
        name: dubbo-demo-consumer
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: dubbo-demo-consumer
      template:
        metadata:
          labels: 
            app: dubbo-demo-consumer
            name: dubbo-demo-consumer
        spec:
          containers:
          - name: dubbo-demo-consumer
            image: harbor.od.com/app/dubbo-demo-consumer:apllo_190120_1815
            ports:
            - containerPort: 20880
              protocol: TCP
            - containerPort: 8080
              protocol: TCP
            env:
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config.od.com
            - name: JAR_BALL
              value: dubbo-client.jar
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600

    注意：增加了env段配置
    注意：docker镜像新版的tag

    应用资源配置清单

    在任意一台k8s运算节点上执行：

    1
    2
    		 
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-web/deployment.yaml
    deployment.extensions/dubbo-demo-consumer configured

    通过Apollo配置中心动态维护项目的配置

    以dubbo-demo-service项目为例，不用修改代码

    实战维护多套dubbo微服务环境

    生产实践

    1. 迭代新需求/修复BUG（编码->提GIT）
    2. 测试环境发版，测试（应用通过编译打包发布至TEST命名空间）
    3. 测试通过，上线（应用镜像直接发布至PROD命名空间）

    系统架构

    • 物理架构
    主机名角色ip
    HDSS7-11.host.com zk-test(测试环境Test) 10.4.7.11
    HDSS7-12.host.com zk-prod(生产环境Prod) 10.4.7.12
    HDSS7-21.host.com kubernetes运算节点 10.4.7.21
    HDSS7-22.host.com kubernetes运算节点 10.4.7.22
    HDSS7-200.host.com 运维主机，harbor仓库 10.4.7.200
    • K8S内系统架构
    环境命名空间应用
    测试环境（TEST） test apollo-config，apollo-admin
    测试环境（TEST） test dubbo-demo-service，dubbo-demo-web
    生产环境（PROD） prod apollo-config，apollo-admin
    生产环境（PROD） prod dubbo-demo-service，dubbo-demo-web
    ops环境（infra） infra jenkins，dubbo-monitor，apollo-portal

    修改/添加域名解析

    DNS主机HDSS7-11.host.com上：

    /var/named/od.com.zone 
    1
    2
    3
    4
    5
    6
    		 
    zk-test 60 IN A 10.4.7.11
    zk-prod 60 IN A 10.4.7.12
    config-test	60 IN A 10.4.7.10
    config-prod	60 IN A 10.4.7.10
    demo-test 60 IN A 10.4.7.10
    demo-prod 60 IN A 10.4.7.10

    Apollo的k8s应用配置

    • 删除app命名空间内应用，创建test命名空间，创建prod命名空间
    • 删除infra命名空间内apollo-configservice，apollo-adminservice应用
    • 数据库内删除ApolloConfigDB，创建ApolloConfigTestDB，创建ApolloConfigProdDB
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    		 
    mysql> drop database ApolloConfigDB;

    mysql> create database ApolloConfigTestDB;
    mysql> use ApolloConfigTestDB;
    mysql> source ./apolloconfig.sql
    mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

    mysql> create database ApolloConfigProdDB;
    mysql> use ApolloConfigProdDB;
    mysql> source ./apolloconfig.sql
    mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
    • 准备apollo-config，apollo-admin的资源配置清单（各2套）

    注：apollo-config/apollo-admin的configmap配置要点

    • Test环境
    1
    2
    3
    4
    5
    6
    		 
    application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config-test.od.com/eureka
    • Prod环境
    1
    2
    3
    4
    5
    6
    		 
    application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config-prod.od.com/eureka
    • 依次应用，分别发布在test和prod命名空间
    • 修改apollo-portal的configmap并重启portal
    1
    2
    3
    		 
    apollo-env.properties: |
        TEST.meta=http://config-test.od.com
        PROD.meta=http://config-prod.od.com

    Apollo的portal配置

    管理员工具

    删除应用、集群、AppNamespace，将已配置应用删除

    系统参数

    • Key

      apollo.portal.envs

    • Value

      TEST,PROD

    查询

    • Value

      TEST,PROD

    保存

    新建dubbo-demo-service和dubbo-demo-web项目

    在TEST/PROD环境分别增加配置项并发布

    发布dubbo微服务

    • 准备dubbo-demo-service和dubbo-demo-web的资源配置清单（各2套）
    • 依次应用，分别发布至app-test和app-prod命名空间
    • 使用dubbo-monitor查验

    互联网公司技术部的日常

    • 产品经理整理需求，需求评审，出产品原型
    • 开发同学夜以继日的开发，提测
    • 测试同学使用Jenkins持续集成，并发布至测试环境
    • 验证功能，通过->待上线or打回->修改代码
    • 提交发版申请，运维同学将测试后的包发往生产环境
    • 无尽的BUG修复（笑cry）
  • 相关阅读:
    技术晨读_20160611
    浏览器退出之后php还会继续执行么？
    大话keepalive
    也说说TIME_WAIT状态
    PHP的错误机制总结
    ASP.NET MVC中使用Unity Ioc Container
    Unity依赖注入使用详解
    小菜学习设计模式（五）—控制反转（Ioc）
    程序员的人性思考（续）
    Delegate、Predicate、Action和Func
  • 原文地址：https://www.cnblogs.com/wangshuyang/p/11648304.html
Copyright © 2020-2023  润新知