简单研究下kubesphere 中devops 的使用。
1. 安装
实际上就是修改安装过程中的 cluster-configuration.yaml 文件的devops.enable 设为true 即可, kebesphere 会自己下载jenkins，我们通过kubesphere 操作流水线最终也会到jenkins 中操作对应的流水线。
参考: https://kubesphere.io/zh/docs/pluggable-components/devops/
安装过程中出现的问题: nfs服务器磁盘空间、k8snode01 磁盘空间、k8snode01 内存问题等等。建议在虚拟机初始化就设置的大一点。
1. 安装成功后查看相关pods、service 如下：
[root@k8smaster01 kubesphere]# kubectl get pods -n kubesphere-devops-system NAME READY STATUS RESTARTS AGE devops-27439710-f4q5f 0/1 Completed 0 8m12s devops-apiserver-7c6774fff5-nmj84 1/1 Running 0 21m devops-controller-98975d478-4xhxn 1/1 Running 0 21m devops-jenkins-64464f495f-4z8br 1/1 Running 1 21m s2ioperator-0 1/1 Running 0 21m [root@k8smaster01 kubesphere]# kubectl get svc -n kubesphere-devops-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE devops-apiserver ClusterIP 10.1.248.197 <none> 9090/TCP 21m devops-jenkins NodePort 10.1.45.197 <none> 80:30180/TCP 21m devops-jenkins-agent ClusterIP 10.1.113.105 <none> 50000/TCP 21m s2ioperator-metrics-service ClusterIP 10.1.161.145 <none> 8080/TCP 21m s2ioperator-trigger-service ClusterIP 10.1.119.55 <none> 8081/TCP 21m webhook-server-service ClusterIP 10.1.46.76 <none> 443/TCP 21m
2. 然后登录kubesphere 控制台看到如下:
2. 登录jenkins
上面可以看到service， 30180 端口可以访问jenkins，在找密码的时候发现其没有密码文件。 最后通过官网发现： 是用kubesphere 的用户体系进行登录。
参考: https://v2-1.docs.kubesphere.io/docs/zh-CN/devops/jenkins-setting/
接下来可以简单的看一下jenkins 控制台界面安装的一些插件等信息。
2. 使用
接下来就是使用kubesphere 的devops 发布一个简单的springboot 项目。 这里使用github作为代码仓库，阿里镜像仓库作为镜像仓库。
官网对其运行过程解释如下：
首先，Jenkins Master 创建一个 Pod 来运行流水线。Kubernetes 创建 Pod 作为 Jenkins Master 的 Agent，该 Pod 会在流水线完成之后销毁。主要流程包括克隆代码、构建和推送镜像以及部署工作负载。
官方提供的两个java 项目：
https://github.com/kubesphere/devops-java-sample
https://github.com/kubesphere/devops-maven-sample 本次测试使用这个项目进行测试
搭建过程参考：
https://kubesphere.io/docs/devops-user-guide/how-to-use/create-a-pipeline-using-jenkinsfile/
1. fork项目到自己的github 仓库
如果网络波动太大的话可以使用国内的gitee，作为代码仓库进行测试。我下面使用gitee。
2. 到kubesphere 进行操作
这里用户都用admin 进行操作，如果想测试kubesphere 的用户体系可以自己创建多个用户进行操作。
1. 到访问控制-》企业空间创建一个企业空间 dev， 用于独立的测试。这个可以理解为大的一个资源隔离的空间。
一个空间可以有多个普通项目和devops 项目。
普通项目可以用来管理独立的kubernetes 资源，包括pod、svc 等，实际kubesphere 后台的操作就是新建了一个对应的namespace用于隔离相应资源，namespace 的名称就是项目名称；
devops 项目可以包含多个jenkins流水线， 用于devops 操作，kubesphere 后台也创建了一个对应的namespace，namespace名称为devops 项目名称加5位随机字母。
2. dev 企业空间创建一个项目，名称为 demo。 查看namespace如下
[root@k8smaster01 ~]# kubectl get ns|grep demo demo59wjg Active 26s
3. 在demo 项目下面建立自己的凭证：
1》gitee 的凭证
2》镜像仓库的凭证
3》kubeconfig 用于kubernetes 使用
4. 本地复制项目Jenkinsfile-online，命名为Jenkinsfile。 原来文件用于备份，新文件用于jenkins 流水线脚本。修改Jenkinsfile 里面内容的凭证信息，最终如下：
1 pipeline { 2 agent { 3 node { 4 label 'maven' 5 } 6 } 7 8 parameters { 9 string(name:'TAG_NAME',defaultValue: '',description:'') 10 } 11 12 environment { 13 DOCKER_CREDENTIAL_ID = 'ali-registery' 14 GITHUB_CREDENTIAL_ID = 'gitee-secret' 15 KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig' 16 REGISTRY = 'registry.cn-hangzhou.aliyuncs.com' 17 DOCKERHUB_NAMESPACE = 'qlq_repository' 18 GITHUB_ACCOUNT = 'qiao-zhi' 19 APP_NAME = 'devops-maven-sample' 20 } 21 22 stages { 23 stage ('checkout scm') { 24 steps { 25 checkout(scm) 26 } 27 } 28 29 stage ('unit test') { 30 steps { 31 container ('maven') { 32 sh 'mvn clean test' 33 } 34 } 35 } 36 37 stage ('build & push') { 38 steps { 39 container ('maven') { 40 sh 'mvn clean package -DskipTests' 41 sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .' 42 withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) { 43 sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' 44 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER' 45 } 46 } 47 } 48 } 49 50 stage('push latest'){ 51 when{ 52 branch 'master' 53 } 54 steps{ 55 container ('maven') { 56 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' 57 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' 58 } 59 } 60 } 61 62 stage('deploy to dev') { 63 when{ 64 branch 'master' 65 } 66 steps { 67 input(id: 'deploy-to-dev', message: 'deploy to dev?') 68 container ('maven') { 69 withCredentials([ 70 kubeconfigFile( 71 credentialsId: env.KUBECONFIG_CREDENTIAL_ID, 72 variable: 'KUBECONFIG') 73 ]) { 74 sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -' 75 } 76 } 77 } 78 } 79 stage('push with tag'){ 80 when{ 81 expression{ 82 return params.TAG_NAME =~ /v.*/ 83 } 84 } 85 steps { 86 container ('maven') { 87 input(id: 'release-image-with-tag', message: 'release image with tag?') 88 withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) { 89 sh 'git config --global user.email "kubesphere@yunify.com" ' 90 sh 'git config --global user.name "kubesphere" ' 91 sh 'git tag -a $TAG_NAME -m "$TAG_NAME" ' 92 sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4' 93 } 94 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' 95 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' 96 } 97 } 98 } 99 stage('deploy to production') { 100 when{ 101 expression{ 102 return params.TAG_NAME =~ /v.*/ 103 } 104 } 105 steps { 106 input(id: 'deploy-to-production', message: 'deploy to production?') 107 container ('maven') { 108 withCredentials([ 109 kubeconfigFile( 110 credentialsId: env.KUBECONFIG_CREDENTIAL_ID, 111 variable: 'KUBECONFIG') 112 ]) { 113 sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -' 114 } 115 } 116 } 117 } 118 } 119 }
和原来官网项目文件相比修改的包括environment 中的凭证的id 和 自己的相关信息； 92 行推送tag 时候的相关的github 信息替换为gitee 服务器信息和用$APP_NAME 获取仓库名称。
简单理解上面相关凭证用于拉代码、推镜像、部署到kubernetes。 REGISTRY 是docker 镜像仓库地址；DOCKERHUB_NAMESPACE 是镜像仓库命名空间；GITHUB_ACCOUNT 用于下面拼接github 代码仓库路径； APP_NAME 是打包名称和镜像名称。
5. 到阿里镜像仓库创建新的镜像仓库，名称为 devops-maven-sample(这里创建为公共仓库，不然拉取镜像会失败， 想解决参考： https://developer.aliyun.com/ask/11155?spm=a2c6h.13706215.ask-content.1.3c732a3aUTxxTA)
6. 到kubesphere demo 项目下新建流水线， 名称为 devops-maven-sample, 然后选择代码仓库，后面默认即可(Jenkinsfile 名称也匹配)：
7. 点到流水线中，然后点击扫描项目，获取git 上面的Jenkinsfile 文件和代码分支，最终的扫描日志如下：
Started by user admin [Sun Mar 06 03:47:01 UTC 2022] Starting branch indexing... > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git ls-remote --symref -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching & pruning origin... Listing remote references... > git config --get remote.origin.url # timeout=10 > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git ls-remote -h -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from origin > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking branches... Checking branch master ‘Jenkinsfile’ found Met criteria No changes detected: master (still at 4b2c785bec6d7c4d2899ec78682b431455529c16) Processed 1 branches [Sun Mar 06 03:47:10 UTC 2022] Finished branch indexing. Indexing took 9.2 sec Finished: SUCCESS
8. 接下来在kubernetes 环境中创建两个namespace， 不创建namespace 在创建kubernetes 资源的时候会报错
创建dev 和 prod 环境对应的namespace
kubectl create ns kubesphere-sample-dev
kubectl create ns kubesphere-sample-prod
9. 然后选择master分支后，点击运行，输入参数v1(该参数用于jenkins 流水线脚本进行判断是否需要推送代码tag和镜像tag)运行流水线，等待其结果：(当我们不输入tag的时候相当于只是部署到开发环境，不打tag等操作)
中间build 过程中，我们全部选择处理
10. 完成后输出如下：
查看日志如下：
Started by user admin > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git --version # 'git version 2.11.0' > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/master Seen 1 remote branch Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task ‘maven-2whbc’ is offline Agent maven-2whbc is provisioned from template maven --- apiVersion: "v1" kind: "Pod" metadata: annotations: {} labels: jenkins: "slave" jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28" jenkins/label: "maven" name: "maven-2whbc" spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: "node-role.kubernetes.io/worker" operator: "In" values: - "ci" weight: 1 containers: - args: - "********" - "maven-2whbc" command: - "jenkins-slave" env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "devops-jenkins-agent.kubesphere-devops-system:50000" - name: "JENKINS_AGENT_NAME" value: "maven-2whbc" - name: "JENKINS_NAME" value: "maven-2whbc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://devops-jenkins.kubesphere-devops-system:80/" image: "jenkins/jnlp-slave:3.27-1" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: memory: "1536Mi" cpu: "500m" requests: memory: "400Mi" cpu: "50m" tty: false volumeMounts: - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "cat" image: "kubesphere/builder-maven:v3.2.0" imagePullPolicy: "IfNotPresent" name: "maven" resources: limits: ephemeral-storage: "10Gi" memory: "8192Mi" cpu: "4000m" requests: ephemeral-storage: "1Gi" memory: "100Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml" name: "config-volume" subPath: "settings.xml" - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false workingDir: "/home/jenkins/agent" nodeSelector: {} restartPolicy: "Never" securityContext: fsGroup: 1000 tolerations: - effect: "NoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" - effect: "PreferNoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" volumes: - hostPath: path: "/var/run/docker.sock" name: "volume-0" - hostPath: path: "/var/data/jenkins_sonar_cache" name: "volume-2" - hostPath: path: "/var/data/jenkins_maven_cache" name: "volume-1" - emptyDir: medium: "" name: "workspace-volume" - configMap: items: - key: "MavenSetting" path: "settings.xml" name: "ks-devops-agent" name: "config-volume" Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 Commit message: "v1" > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (checkout scm) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Fetching changes from the remote Git repository Fetching without tags > git rev-parse --is-inside-work-tree # timeout=10 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) Commit message: "v1" [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (unit test) [Pipeline] container [Pipeline] { [Pipeline] sh > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 + mvn clean test [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.kubesphere.devops.HelloWorldControllerTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.872 s [INFO] Finished at: 2022-03-06T04:25:53Z [INFO] ------------------------------------------------------------------------ [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (build & push) [Pipeline] container [Pipeline] { [Pipeline] sh + mvn clean package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample --- [INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.580 s [INFO] Finished at: 2022-03-06T04:26:16Z [INFO] ------------------------------------------------------------------------ [Pipeline] sh + docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 . Sending build context to Docker daemon 17.52MB Step 1/4 : FROM java:openjdk-8-jre-alpine ---> fdc893b19a14 Step 2/4 : WORKDIR /home ---> Using cache ---> 1e0dc3b20420 Step 3/4 : COPY target/*.jar /home ---> fd0896260f0e Step 4/4 : ENTRYPOINT java -jar *.jar ---> Running in 27d46c26a862 Removing intermediate container 27d46c26a862 ---> 8867ca53b5ca Successfully built 8867ca53b5ca Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 [Pipeline] withCredentials Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD [Pipeline] { [Pipeline] sh + echo **** + docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Pushed SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push latest) [Pipeline] container [Pipeline] { [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 78075328e0da: Layer already exists 20dd87a4c2ab: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to dev) [Pipeline] input deploy to dev? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample-dev configured service/ks-sample-dev unchanged [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push with tag) [Pipeline] container [Pipeline] { [Pipeline] input release image with tag? Proceed or Abort Approved by admin [Pipeline] withCredentials Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD [Pipeline] { [Pipeline] sh + git config --global user.email kubesphere@yunify.com [Pipeline] sh + git config --global user.name kubesphere [Pipeline] sh + git tag -a v1 -m v1 [Pipeline] sh + git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4 remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m To http://gitee.com/****/devops-maven-sample.git * [new tag] v1 -> v1 [Pipeline] } [Pipeline] // withCredentials [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to production) [Pipeline] input deploy to production? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample created service/ks-sample created [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
1》jenkins 查看日志如下
日志如下：
Started by user admin > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git --version # 'git version 2.11.0' > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/master Seen 1 remote branch Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task ‘maven-2whbc’ is offline Agent maven-2whbc is provisioned from template maven --- apiVersion: "v1" kind: "Pod" metadata: annotations: {} labels: jenkins: "slave" jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28" jenkins/label: "maven" name: "maven-2whbc" spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: "node-role.kubernetes.io/worker" operator: "In" values: - "ci" weight: 1 containers: - args: - "********" - "maven-2whbc" command: - "jenkins-slave" env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "devops-jenkins-agent.kubesphere-devops-system:50000" - name: "JENKINS_AGENT_NAME" value: "maven-2whbc" - name: "JENKINS_NAME" value: "maven-2whbc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://devops-jenkins.kubesphere-devops-system:80/" image: "jenkins/jnlp-slave:3.27-1" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: memory: "1536Mi" cpu: "500m" requests: memory: "400Mi" cpu: "50m" tty: false volumeMounts: - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "cat" image: "kubesphere/builder-maven:v3.2.0" imagePullPolicy: "IfNotPresent" name: "maven" resources: limits: ephemeral-storage: "10Gi" memory: "8192Mi" cpu: "4000m" requests: ephemeral-storage: "1Gi" memory: "100Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml" name: "config-volume" subPath: "settings.xml" - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false workingDir: "/home/jenkins/agent" nodeSelector: {} restartPolicy: "Never" securityContext: fsGroup: 1000 tolerations: - effect: "NoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" - effect: "PreferNoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" volumes: - hostPath: path: "/var/run/docker.sock" name: "volume-0" - hostPath: path: "/var/data/jenkins_sonar_cache" name: "volume-2" - hostPath: path: "/var/data/jenkins_maven_cache" name: "volume-1" - emptyDir: medium: "" name: "workspace-volume" - configMap: items: - key: "MavenSetting" path: "settings.xml" name: "ks-devops-agent" name: "config-volume" Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 Commit message: "v1" > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (checkout scm) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Fetching changes from the remote Git repository Fetching without tags > git rev-parse --is-inside-work-tree # timeout=10 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) Commit message: "v1" [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (unit test) [Pipeline] container [Pipeline] { [Pipeline] sh > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 + mvn clean test [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.kubesphere.devops.HelloWorldControllerTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.872 s [INFO] Finished at: 2022-03-06T04:25:53Z [INFO] ------------------------------------------------------------------------ [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (build & push) [Pipeline] container [Pipeline] { [Pipeline] sh + mvn clean package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample --- [INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.580 s [INFO] Finished at: 2022-03-06T04:26:16Z [INFO] ------------------------------------------------------------------------ [Pipeline] sh + docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 . Sending build context to Docker daemon 17.52MB Step 1/4 : FROM java:openjdk-8-jre-alpine ---> fdc893b19a14 Step 2/4 : WORKDIR /home ---> Using cache ---> 1e0dc3b20420 Step 3/4 : COPY target/*.jar /home ---> fd0896260f0e Step 4/4 : ENTRYPOINT java -jar *.jar ---> Running in 27d46c26a862 Removing intermediate container 27d46c26a862 ---> 8867ca53b5ca Successfully built 8867ca53b5ca Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 [Pipeline] withCredentials Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD [Pipeline] { [Pipeline] sh + echo **** + docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Pushed SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push latest) [Pipeline] container [Pipeline] { [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 78075328e0da: Layer already exists 20dd87a4c2ab: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to dev) [Pipeline] input deploy to dev? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample-dev configured service/ks-sample-dev unchanged [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push with tag) [Pipeline] container [Pipeline] { [Pipeline] input release image with tag? Proceed or Abort Approved by admin [Pipeline] withCredentials Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD [Pipeline] { [Pipeline] sh + git config --global user.email kubesphere@yunify.com [Pipeline] sh + git config --global user.name kubesphere [Pipeline] sh + git tag -a v1 -m v1 [Pipeline] sh + git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4 remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m To http://gitee.com/****/devops-maven-sample.git * [new tag] v1 -> v1 [Pipeline] } [Pipeline] // withCredentials [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to production) [Pipeline] input deploy to production? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample created service/ks-sample created [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
2》 kubernetes 环境查看相关的资源
[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-devops-dev No resources found in kubesphere-devops-dev namespace. [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-dev NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample-dev 1/1 1 1 12h NAME READY STATUS RESTARTS AGE pod/ks-sample-dev-5dc9786dcc-6f9fx 1/1 Running 0 5m48s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample-dev NodePort 10.1.66.88 <none> 8080:30861/TCP 14h [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-prod NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample 2/2 2 2 4m57s NAME READY STATUS RESTARTS AGE pod/ks-sample-68b68dfbc6-dpkwk 1/1 Running 0 4m57s pod/ks-sample-68b68dfbc6-qs2sw 1/1 Running 0 4m57s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample NodePort 10.1.21.72 <none> 8080:30961/TCP 4m57s
3》gitee 查看打的tag 信息
4》阿里镜像仓库查看版本信息如下：
5》测试访问
xx@xx MINGW64 /d/study/devops-maven-sample (master) $ curl http://192.168.13.107:30861 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2 100 2 0 0 166 0 --:--:-- --:--:-- --:--:-- 181v1 xx@xx MINGW64 /d/study/devops-maven-sample (master) $ curl http://192.168.13.107:30961 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2 100 2 0 0 68 0 --:--:-- --:--:-- --:--:-- 71v1
11. 总结
(0) 前置
其实通过kubesphere 流水线操作，最终还是都会到Jenkins中进行操作，其操作和自定义流水线一样。大致就是打代码、测试代码、打包、制作镜像并打tag、传送到镜像仓库、创建kubernetes 相关资源。在jenkins 执行过程中，会创建一个代理pod，比如：
[root@k8smaster01 ~]# kubectl get pods -A| grep maven kubesphere-devops-worker maven-cfqfb 2/2 Running 0 47s kubesphere-devops-worker maven-gzjdm 1/2 Error 0 18h
其中git 拉取代码、maven打包都是在下面pods 操作的。进入一个pod，查看其环境如下：
(1)Jenkins file文件查看分析
pipeline { agent { node { label 'maven' } } parameters { string(name:'TAG_NAME',defaultValue: '',description:'') } environment { DOCKER_CREDENTIAL_ID = 'ali-registery' GITHUB_CREDENTIAL_ID = 'gitee-secret' KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig' REGISTRY = 'registry.cn-hangzhou.aliyuncs.com' DOCKERHUB_NAMESPACE = 'qlq_repository' GITHUB_ACCOUNT = 'qiao-zhi' APP_NAME = 'devops-maven-sample' } stages { stage ('checkout scm') { steps { checkout(scm) } } stage ('unit test') { steps { container ('maven') { sh 'mvn clean test' } } } stage ('build & push') { steps { container ('maven') { sh 'mvn clean package -DskipTests' sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .' withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) { sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER' } } } } stage('push latest'){ when{ branch 'master' } steps{ container ('maven') { sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' } } } stage('deploy to dev') { when{ branch 'master' } steps { input(id: 'deploy-to-dev', message: 'deploy to dev?') container ('maven') { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) { sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -' } } } } stage('push with tag'){ when{ expression{ return params.TAG_NAME =~ /v.*/ } } steps { container ('maven') { input(id: 'release-image-with-tag', message: 'release image with tag?') withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) { sh 'git config --global user.email "kubesphere@yunify.com" ' sh 'git config --global user.name "kubesphere" ' sh 'git tag -a $TAG_NAME -m "$TAG_NAME" ' sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4' } sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' } } } stage('deploy to production') { when{ expression{ return params.TAG_NAME =~ /v.*/ } } steps { input(id: 'deploy-to-production', message: 'deploy to production?') container ('maven') { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) { sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -' } } } } } }
前面定义了一些全局环境变量，按阶段进行分析
1》checkout scm 拉取代码
2》unit test 执行单元测试
3》build & push: 执行mvn打包， 然后基于Dockerfile-online 构造本地镜像，镜像的版本用代码分支和buildnumber 做区分，然后将镜像推送代码镜像仓库
4》push latest： 当代码分支是master 的时候，将3》的镜像打包为latest版本的镜像，然后推到镜像仓库
5》deploy to dev：当代码分支是master 的时候，相当于是执行 deploy/dev-all-in-one/devops-sample.yaml文件，相当于到kubernetes 创建资源
6》push with tag：当我们传递的参数是vxxx的时候，将代码打上相应的标签并推送到代码仓库；将镜像也打上相应的标签并推送到相应的镜像仓库
7》deploy to production：当我们传递的参数是vxxx的时候， 需要我们验证下是否需要将相关资源创建到kubernetes环境的prod 仓库，使用的文件是deploy/prod-all-in-one/devops-sample.yaml。
envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -
envsubst 是用shell格式字符串中的值替换环境变量。要替换的变量应位于${var}或$var格式。例如
[root@k8smaster01 ~]# export MYKEY=123456 [root@k8smaster01 ~]# echo $MYKEY 123456 [root@k8smaster01 ~]# cat test.txt $MYKEY [root@k8smaster01 ~]# envsubst < test.txt 123456
(2) 几个重要kubernetes 文件如下：
deploy/dev-all-in-one/devops-sample.yaml：
--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kubesphere component: ks-sample-dev tier: backend name: ks-sample-dev namespace: kubesphere-sample-dev spec: progressDeadlineSeconds: 600 replicas: 1 selector: matchLabels: app: kubesphere component: ks-sample-dev tier: backend template: metadata: labels: app: kubesphere component: ks-sample-dev tier: backend spec: containers: - env: - name: CACHE_IGNORE value: js|html - name: CACHE_PUBLIC_EXPIRATION value: 3d image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER readinessProbe: httpGet: path: / port: 8080 timeoutSeconds: 10 failureThreshold: 30 periodSeconds: 5 imagePullPolicy: Always name: ks-sample ports: - containerPort: 8080 protocol: TCP resources: limits: cpu: 300m memory: 600Mi requests: cpu: 100m memory: 100Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: labels: app: kubesphere component: ks-sample-dev name: ks-sample-dev namespace: kubesphere-sample-dev spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30861 selector: app: kubesphere component: ks-sample-dev tier: backend sessionAffinity: None type: NodePort
deploy/prod-all-in-one/devops-sample.yaml
--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kubesphere component: ks-sample tier: backend name: ks-sample namespace: kubesphere-sample-prod spec: progressDeadlineSeconds: 600 replicas: 2 selector: matchLabels: app: kubesphere component: ks-sample tier: backend strategy: rollingUpdate: maxSurge: 100% maxUnavailable: 100% type: RollingUpdate template: metadata: labels: app: kubesphere component: ks-sample tier: backend spec: containers: - env: - name: CACHE_IGNORE value: js|html - name: CACHE_PUBLIC_EXPIRATION value: 3d image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME readinessProbe: httpGet: path: / port: 8080 timeoutSeconds: 10 failureThreshold: 30 periodSeconds: 5 imagePullPolicy: Always name: ks ports: - containerPort: 8080 protocol: TCP resources: limits: cpu: 300m memory: 600Mi requests: cpu: 100m memory: 100Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: labels: app: kubesphere component: ks-sample name: ks-sample namespace: kubesphere-sample-prod spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30961 selector: app: kubesphere component: ks-sample tier: backend sessionAffinity: None type: NodePort
代码仓库和修改后配置文件：https://gitee.com/Qiao-Zhi/devops-maven-sample
补充： 在安装过程中遇到一些磁盘不足，报错信息如下:
Warning Evicted 49m kubelet The node had condition: [DiskPressure].
解决参考：
https://blog.51cto.com/riverxyz/2758421
vm中对centos 扩容: https://ld246.com/article/1566021346577