• Kubernetes之动态Jenkins slave


    一、前提

    本次实践前,需已完成以下过程:

    1、搭建好一个Kubernetes集群(本实践为单节点集群),网上参考较多,不赘述。

    2、选取kubernetes集群外的一台服务器安装 NFS服务端,并在集群内每个节点安装 NFS客户端;
    NFS服务端所在的服务器IP为 10.141.211.178,记为 nfs server;而集群master服务器,记为 master;
    (1) nfs server创建存储目录 /data/k8s/ 并执行命令:chmod 755 /data/k8s/, 并关闭防火墙
    (2) nfs server安装NFS,执行: yum -y install nfs-utils rpcbind
    再配置NFS,执行:vi /etc/exports,在该文件内添加内容:/data/k8s *(rw,sync,no_root_squash)
    然后启动NFS服务,执行:

        # systemctl start rpcbind && systemctl enable rpcbind
        # systemctl start nfs && systemctl enable nfs
    

    ​ (3) master同样安装并启动NFS,执行:

        # yum -y install nfs-utils rpcbind
        # systemctl start rpcbind && systemctl enable rpcbind
        # systemctl start nfs && systemctl enable nfs
    

    ​ 再执行:showmount -e 10.141.211.178,可看到共享目录 /data/k8s

    二、集群安装Jenkins

    Jenkins master的安装,需要将数据持久化。可以利用NFS作为存储资源,创建PVC对象来挂载。PV/PVC配置文件pvc.yaml如下:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: opspv
    spec:
      capacity:
        storage: 20Gi
      accessModes:
      - ReadWriteMany
      persistentVolumeReclaimPolicy: Delete
      nfs:
        server: 10.141.211.178          #注意:此处为NFS服务器的地址
        path: /data/k8s
    
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: opspvc
      namespace: kube-ops
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 20Gi
    

    同时,对于即将创建的Jenkins master资源对象,需要授予其一些权限,比如增删改查等。相应的配置文件rbac.yaml如下:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins
      namespace: kube-ops
    
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: jenkins
    rules:
      - apiGroups: ["extensions", "apps"]
        resources: ["deployments"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/exec"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/log"]
        verbs: ["get","list","watch"]
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: jenkins
      namespace: kube-ops
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: jenkins
    subjects:
      - kind: ServiceAccount
        name: jenkins
        namespace: kube-ops
    

    然后,基于jenkins/jenkins:lts 镜像创建jenkins master镜像,配置文件jenkins.yaml 如下:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: jenkins
      namespace: kube-ops
    spec:
      template:
        metadata:
          labels:
            app: jenkins
        spec:
          terminationGracePeriodSeconds: 10
          serviceAccount: jenkins
          containers:
          - name: jenkins
            image: jenkins/jenkins:lts
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
            resources:
              limits:
                cpu: 1000m
                memory: 1Gi
              requests:
                cpu: 500m
                memory: 512Mi
            livenessProbe:
              httpGet:
                path: /login
                port: 8080
              initialDelaySeconds: 60
              timeoutSeconds: 5
              failureThreshold: 12
            readinessProbe:
              httpGet:
                path: /login
                port: 8080
              initialDelaySeconds: 60
              timeoutSeconds: 5
              failureThreshold: 12
            volumeMounts:
            - name: jenkinshome
              subPath: jenkins
              mountPath: /var/jenkins_home
            env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
          securityContext:
            fsGroup: 1000
          volumes:
          - name: jenkinshome
            persistentVolumeClaim:
              claimName: opspvc
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: jenkins
      namespace: kube-ops
      labels:
        app: jenkins
    spec:
      selector:
        app: jenkins
      type: NodePort
      ports:
      - name: web
        port: 8080
        targetPort: web
        nodePort: 30002
      - name: agent
        port: 50000
        targetPort: agent
    

    最后,在一个目录内分别创建以上3个文件,执行命令如下:

    # kubectl create namespace kube-ops
    # kubectl create -f pvc.yaml
    # kubectl create -f rbac.yaml
    # kubectl create -f jenkins.yaml(此步执行会出现文件权限问题,解决办法为:
              先在nfs server服务器执行:chown -R 1000 /data/k8s/jenkins
              然后在master执行:kubectl delete -f jenkins.yaml
              kubectl create -f jenkins.yaml  )
    

    此时,我们通过命令kubectl -n kube-ops get pod可以查看到jenkins已成功创建。

    三、Jenkins配置动态slave

    1、初始化Jenkins配置

    浏览器打开masterIP:30002,如下:

    其中的管理员密码,我们既可以进入容器内对应的目录查看,也可以在nfs server服务器上执行命令:
    cat /data/k8s/jenkins/secrets/initialAdminPassword 来查看;然后选择安装推荐的插件,如下:

    然后添加管理员账户即可进入Jenkins界面。

    2、配置jenkins slave

    (1) 安装Kubernetes插件
    进入 Manage Jenkins—>Manage Plugins—>可选插件(Available)—>Kubernetes plugin勾选,直接安装即可。
    (2) 配置Kubernetes插件
    点击Manage Jenkins—>Configure System—>云—>新增一个云—>Kubernetes,如下:

    然后配置如下:
    先注意 名称默认为kubernetes,然后 Kubernetes地址 填写https://kubernetes.default.svc.cluster.local,命名空间为kube-ops;接着点击右边的 连接测试 按钮,如果显示Connection test successful,表示Jenkins可以和Kubernetes集群正常通信了。最后,在Jenkins地址,填入:http://服务名.kube-ops.svc.cluster.local:8080,如下所示:

    (3) 配置 Kubernetes Pod Template
    关于 Kubernetes Pod Template部分的配置,其实就是对jenkins slave的配置。具体配置如下:

    图中标记的地方较为重要,不要填错。其中标签列表部分 后面仍有用到;Docker 镜像部分,是本人基于 cnych/jenkins:jnlp6 镜像基础上继续定制的镜像,包含maven、docker、docker-compose、kubectl等工具。
    另外,添加卷如下:

    添加这两个 Host Path Volume,是为了更好地在jenkins slave容器中使用docker 和 kubectl 工具,所以挂载了宿主机的部分目录。
    然后,设置Service Account如下:

    最后,点击 保存 即可。

    3.测试jenkins slave

    首先新建一个 名为test 的 Freestyle project 项目,其配置如下:

    这里的标签表达式,正是Kubernetes Pod Template的标签列表的内容。
    然后,增加构建步骤—>执行shell,如下:

    具体shell如下:

    echo "测试 Kubernetes 动态生成 jenkins slave"
    
    echo "===========mvn==========="
    mvn --version
    echo $PATH
    
    echo "==============docker in docker==========="
    which docker
    docker version
    
    echo "==============docker-compose==========="
    docker-compose version
    
    echo "=============kubectl============="
    kubectl get pods
    

    保存之后,点击 立即构建。控制台输出如下:

    Started by user admin
    Running as SYSTEM
    Agent jnlp-slave-pk06f is provisioned from template Kubernetes Pod Template
    ---
    apiVersion: "v1"
    kind: "Pod"
    metadata:
      annotations: {}
      labels:
        jenkins: "slave"
        jenkins/jnlp-slave: "true"
      name: "jnlp-slave-pk06f"
    spec:
      containers:
      - env:
        - name: "JENKINS_SECRET"
          value: "********"
        - name: "JENKINS_AGENT_NAME"
          value: "jnlp-slave-pk06f"
        - name: "JENKINS_NAME"
          value: "jnlp-slave-pk06f"
        - name: "JENKINS_AGENT_WORKDIR"
          value: "/home/jenkins/agent"
        - name: "JENKINS_URL"
          value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
        image: "zhongyuanzhao000/jenkins-slave:jnlp"
        imagePullPolicy: "IfNotPresent"
        name: "jnlp"
        resources:
          limits: {}
          requests: {}
        securityContext:
          privileged: false
        tty: true
        volumeMounts:
        - mountPath: "/var/run/docker.sock"
          name: "volume-0"
          readOnly: false
        - mountPath: "/root/.kube"
          name: "volume-1"
          readOnly: false
        - mountPath: "/home/jenkins/agent"
          name: "workspace-volume"
          readOnly: false
        workingDir: "/home/jenkins/agent"
      nodeSelector: {}
      restartPolicy: "Never"
      serviceAccount: "jenkins"
      volumes:
      - hostPath:
          path: "/var/run/docker.sock"
        name: "volume-0"
      - hostPath:
          path: "/root/.kube"
        name: "volume-1"
      - emptyDir:
          medium: ""
        name: "workspace-volume"
    
    Building remotely on jnlp-slave-pk06f (jnlp-slave) in workspace /home/jenkins/agent/workspace/test
    [test] $ /bin/sh -xe /tmp/jenkins3820575614440094591.sh
    + echo 测试 Kubernetes 动态生成 jenkins slave
    测试 Kubernetes 动态生成 jenkins slave
    + echo ===========mvn===========
    ===========mvn===========
    + mvn --version
    Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
    Maven home: /usr/local
    Java version: 1.8.0_212, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
    Default locale: zh_CN, platform encoding: UTF-8
    OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
    + echo /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    + echo ==============docker in docker===========
    ==============docker in docker===========
    + which docker
    /usr/local/bin/docker
    + docker version
    Client: Docker Engine - Community
     Version:           18.09.8
     API version:       1.39
     Go version:        go1.10.8
     Git commit:        0dd43dd87f
     Built:             Wed Jul 17 17:38:58 2019
     OS/Arch:           linux/amd64
     Experimental:      false
    
    Server: Docker Engine - Community
     Engine:
      Version:          18.09.7
      API version:      1.39 (minimum version 1.12)
      Go version:       go1.10.8
      Git commit:       2d0083d
      Built:            Thu Jun 27 17:26:28 2019
      OS/Arch:          linux/amd64
      Experimental:     false
    + echo ==============docker-compose===========
    ==============docker-compose===========
    + docker-compose version
    docker-compose version 1.23.2, build 1110ad01
    docker-py version: 3.6.0
    CPython version: 3.6.7
    OpenSSL version: OpenSSL 1.1.0f  25 May 2017
    + echo =============kubectl=============
    =============kubectl=============
    + kubectl get pods
    NAME                       READY   STATUS    RESTARTS   AGE
    jenkins-575b84fb7b-59s5h   1/1     Running   0          29h
    jnlp-slave-pk06f           1/1     Running   0          7s
    Finished: SUCCESS
    

    其中,你可以发现jenkins创建了jnlp-slave-pk06f的slave对象;而当该任务执行完之后,你在master上获取pod就会发现 jnlp-slave-pk06f 这个slave自动消失了,这就是动态jenkins slave的简单体现。

    四、参考

    本次实践得益于诸多运维大神对于知识的不吝分享,十分感谢!!!
    具体参考的博客或指南如下:
    基于 Jenkins 的 CI/CD (一) 强烈推荐阳明老师的博客
    kubernetes实践:安装jenkins slave
    kubernetes Jenkins gitlab搭建CI/CD环境 (二)
    01 [从这里开始]Jenkins CI解决方案

  • 相关阅读:
    关于MySQL 最后一部安装阶段无法响应的问题
    为PHP 启用Mysql 的dll
    webcast终于改版了
    有意思的oracle转义字符
    Telnet协议详解及使用C# 用Socket 编程来实现Telnet协议
    我被百度悲剧了
    阻止 http://3b3.org/c.js恶意SQL注入
    驱动研究日记-链表+后备链表
    C# 执行汇编类
    Tuning and Slimming JBossAS
  • 原文地址:https://www.cnblogs.com/zhongyuanzhao000/p/11430183.html
Copyright © 2020-2023  润新知