使用Jenkins Operator在k8s中部署Jenkins

目录

• 1、Jenkins Operator介绍
• 2、Jenkins Operator的架构和设计
• 3、使用Operator部署Jenkins
  • 3.1 前提条件
  • 3.2 获取并创建CRD
  • 3.3 部署Jenkins Operator
  • 3.4 部署Jenkins

1、Jenkins Operator介绍

Kubernetes Operator是一种特定于应用的控制器，可扩展Kubernetes API的功能，来代表Kubernetes用户创建、配置和管理复杂应用的实例

这里对Operator的相关介绍就不多赘述了，主要还是回到Jenkins Operator这个话题

基于k8s上Jenkins的常规安装是使用yaml资源清单，更为方便一点的是helm chart，但是常常我们在安装后还需要做很多的动作。例如插件问题，这尽管可以通过Configuration as Code的方式来解决，根据个人实际经验来看，还是存在一定几率会因主镜像版本、环境等存在诸多不可预知的问题。当然不同的部署方式都各有利弊，大家根据实际情况选择即可。

直到官方支持Jenkins可以在k8s中通过Operator方式部署，在4月中旬，Jenkins blog说道：Jenkins Operator 正式成为了 Jenkins 的子项目，填补了Jenkins与Kubernetes间的缝隙。也就是说，最初由（个人）三方团队编写的Jenkins Operator被Jenkins官方认可了

参考官方说明，Jenkins Operator可以帮我们解决以下问题：

• 安装指定版本的插件

  即使最新版本插件不兼容或具备安全漏洞，还是为了插件稳定性而使用（因为常常会出现我们通过一键升级插件导致很多问题而去手动安装旧版本插件的情况）

• 更好的自定义配置

  包含在安装指定版本插件时指定插件配置等声明式配置

• 开箱即用的安全配置

• 可灵活调整的debug错误调试

• 备份和还原作业历史记录

  ......

2、Jenkins Operator的架构和设计

参考Jenkins Operator Architecture and design

Jenkins Operator的设计包含以下概念

• 监视清单的任何更改，并根据已部署的自定义资源清单维护所需的状态
• 实现主reconciliation循环，由两个较小的reconciliation循环：base和user

Base reconciliation 循环负责监听Jenkins基础配置：

• 确认清单-监听清单中发生的任何更改
• 确保Jenkins Pod状态，创建和验证Jenkins Server Pod的状态
• 确认Jenkins的配置，包括安全加固、初始化配置等
• 确认Jenkins API token，生成token并初始化Jenkins Client

User reconciliation循环负责协调用户提供的配置：

• 确保恢复任务，创建恢复任务，并确保恢复已成功执行
• 确保Seed Jobs，创建Seed Jobs并确保所有这些工作都已成功执行
• 确保用户配置，执行用户提供的配置，如groovy脚本，配置为代码或插件
• 确保备份任务，创建备份任务并确保备份成功

Operator状态
Operator状态保存在自定义资源状态部分中，该部分用于存储Operator管理的任何配置事件或Job状态

即使操作者或Jenkins重新启动，它也能帮助保持或恢复所需的状态

3、使用Operator部署Jenkins

3.1 前提条件

参考Jenkins Operator官方文档，需要有一个1.11+版本的Kubernetes集群，这里我的环境如下

# kubectl version -o yaml
clientVersion:
  buildDate: "2020-12-08T17:59:43Z"
  compiler: gc
  gitCommit: af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38
  gitTreeState: clean
  gitVersion: v1.20.0
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: darwin/amd64
serverVersion:
  buildDate: "2021-01-13T13:20:00Z"
  compiler: gc
  gitCommit: faecb196815e248d3ecfb03c680a4507229c2a56
  gitTreeState: clean
  gitVersion: v1.20.2
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: linux/amd64

3.2 获取并创建CRD

获取yaml并创建crd，当然也可以通过直接apply远程地址，这里先将其保存到本地

# wget -c https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkins_v1alpha2_jenkins_crd.yaml
# kubectl apply -f jenkins_v1alpha2_jenkins_crd.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/jenkins.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/jenkinsimages.jenkins.io created

3.3 部署Jenkins Operator

有以下两种方式部署Jenkins Operator

• 使用yaml一键安装，默认将安装在default命名空间下

  # kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/all-in-one-v1alpha2.yaml
  # kubectl get pods -w
  

• 使用helm并自定义安装，依赖helm在v3以上版本

创建ns

# kubectl create ns jenkins

添加helm仓库并获取chart

# helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart
"jenkins" has been added to your repositories
# helm pull jenkins/jenkins-operator
# tar xf jenkins-operator-0.4.3.tgz && cd jenkins-operator 

修改value yaml部分内容，可以定义关于jenkins实例、operator deployment、backup备份相关、Configuration配置相关字段

• 指定ns
• 指定插件
• 默认情况只持久化了备份卷，这里将数据卷也做持久化，sc使用csi-rbd-sc
• 默认开启configurationAsCode，并通过configmap和secret注入

jenkins:
...
  namespace: jenkins
...
  basePlugins:
    - name: kubernetes
      version: "1.28.6"
    - name: workflow-job
      version: "2.40"
    - name: workflow-aggregator
      version: "2.6"
    - name: git
      version: "4.5.0"
    - name: job-dsl
      version: "1.77"
    - name: configuration-as-code
      version: "1.46"
    - name: kubernetes-credentials-provider
      version: "0.15"
  plugins:
    - name: simple-theme-plugin
      version: "0.6"
#  plugins: []
...
  # volumes used by Jenkins
  # By default, we are only using backup
  volumes:
    - name: backup # PVC volume where backups will be stored
      persistentVolumeClaim:
        claimName: jenkins-backup

  # volumeMounts are mounts for Jenkins pod
  volumeMounts: []
...
  backup:
    # enabled is enable/disable switch for backup feature
    # By default the feature is enabled
    enabled: true

    # image used by backup feature
    # By default using prebuilt backup PVC image by VirtusLab
    image: virtuslab/jenkins-operator-backup-pvc:v0.1.0

    # containerName is backup container name
    containerName: backup

    # interval defines how often make backup in seconds
    interval: 30

    # makeBackupBeforePodDeletion when enabled will make backup before pod deletion
    makeBackupBeforePodDeletion: true

    # backupCommand is backup container command
    backupCommand:
      - /home/user/bin/backup.sh

    # restoreCommand is backup restore command
    restoreCommand:
      - /home/user/bin/restore.sh

    getLatestAction:
      - /home/user/bin/get-latest.sh

    # pvc is Persistent Volume Claim Kubernetes resource
    pvc:
      # enabled is enable/disable switch for PVC
      enabled: true

      # size is size of PVC
      size: 5Gi

      # className is storageClassName for PVC
      # See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1 for more details
      className: "csi-rbd-sc"

    # env contains container environment variables
    # PVC backup provider handles these variables:
    # BACKUP_DIR - path for storing backup files (default: "/backup")
    # JENKINS_HOME - path to jenkins home (default: "/jenkins-home")
    # BACKUP_COUNT - define how much recent backups will be kept
    env:
      - name: BACKUP_DIR
        value: /backup
      - name: JENKINS_HOME
        value: /jenkins-home
      - name: BACKUP_COUNT
        value: "3" # keep only the 3 most recent backups

    # volumeMounts holds the mount points for volumes
    volumeMounts:
      - name: jenkins-home
        mountPath: /jenkins-home # Jenkins home volume
      - mountPath: /backup # backup volume
        name: backup
...
  configuration:
    configurationAsCode: {}
    #    - configMapName: jenkins-casc
    #      content: {}
    groovyScripts: {}
    #    - configMapName: jenkins-gs
    #      content: {}

    # secretRefName of existing secret (previously created)
    secretRefName: ""

    # secretData creates new secret if secretRefName is empty and fills with data provided in secretData
    secretData: {}

执行安装

# helm install jenkins jenkins-operator -n jenkins --values ./jenkins-operator/values.yaml
NAME: jenkins
LAST DEPLOYED: Sun May 16 19:42:32 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch Jenkins instance being created:
$ kubectl --namespace jenkins get pods -w

2. Get Jenkins credentials:
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d

3. Connect to Jenkins (actual Kubernetes cluster):
$ kubectl --namespace jenkins port-forward jenkins-jenkins 8080:8080

Now open the browser and enter http://localhost:8080

检查创建的operator

# kubectl get pods -n jenkins
NAME                                       READY   STATUS    RESTARTS   AGE
jenkins-jenkins                            1/2     Running   0          44s
jenkins-jenkins-operator-996887c4b-wftz2   1/1     Running   0          1m29s
# kubectl -n jenkins get jenkins        
NAME      AGE
jenkins   70s

3.4 部署Jenkins

一旦上面的Jenkins Operator部署后启动并正常运行，就自动会部署一个Jenkins实例Pod了

实际上可以看到，通过Jenkins Operator部署的Jenkins的控制器不是场景k8s自带的三大控制器，而是由operator自己管控

观察operator的日志如下

# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T11:59:05.017Z        INFO    controller-jenkins      jenkins/jenkins_controller.go:432       Setting default Jenkins API settings    {"cr": "jenkins"}
2021-05-16T11:59:05.073Z        INFO    controller-jenkins      jenkins/handler.go:88   *v1alpha2.Jenkins/jenkins has been updated      {"cr": "jenkins"}
2021-05-16T11:59:06.568Z        INFO    controller-jenkins      base/pod.go:159 Creating a new Jenkins Master Pod jenkins/jenkins-jenkins       {"cr": "jenkins"}

观察jenkins pod中jenkins master的日志如下，正在下载插件（此步骤稍慢）

# kubectl -n jenkins logs -f jenkins-jenkins -c jenkins-master
...
 > bootstrap4-api depends on font-awesome-api:5.15.2-2,jquery3-api:3.5.1-3,popper-api:1.16.1-2
Downloading plugin: font-awesome-api from https://updates.jenkins.io/dynamic-2.263//latest/font-awesome-api.hpi
Downloading plugin: jquery3-api from https://updates.jenkins.io/dynamic-2.263//latest/jquery3-api.hpi
Downloading plugin: popper-api from https://updates.jenkins.io/dynamic-2.263//latest/popper-api.hpi

如果在有限时间（健康检查时间）内没有下载成功，这通常是由于网络原因引起的，Operator会中断该Pod并重新创建

# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T12:09:42.854Z        INFO    controller-jenkins      base/reconcile.go:370   Container 'jenkins-master' is terminated, status '{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:137,Signal:0,Reason:Error,Message:,StartedAt:2021-05-16 12:05:54 +0000 UTC,FinishedAt:2021-05-16 12:09:42 +0000 UTC,ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:jenkins/jenkins:2.263.2-lts-alpine ImageID:docker-pullable://jenkins/jenkins@sha256:496142509b7d3e3f22f5cdc81b1d1322db61ec929d34dfd66b9ec3257bca13e5 ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da Started:0xc0005aa2c6}' {"cr": "jenkins"}

可行的一个解决办法是将value.yaml中的健康检查时间微调或者临时去掉健康检查，并helm更新让其正常启动并持久化后再次恢复，或者新创建一个Jenkins控制器将其覆盖

# helm -n jenkins upgrade jenkins jenkins-operator --values ./jenkins-operator/values.yaml 

最终直到看见这样的日志，就表示Jenkins启动成功了

2021-05-16 13:26:14.221+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@295b7e33]: org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75
2021-05-16 13:26:14.223+0000 [id=28]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75: defining beans [filter,legacy]; root of factory hierarchy
2021-05-16 13:26:14.489+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-05-16 13:26:14.767+0000 [id=20]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running

到这里，通过Jenkins Operator部署Jenkins就完成了（尽管看上去也没多少比helm或传统方式部署的优势），其实Jenkins Operator还有更为好用的的其他功能，后续再介绍。

See you ~