在k8s中部署jenkins
1.在k8s中部署jenkins
2. Jenkins在Kubernetes中动态创建代理
Jenkins Master/Slave架构
3. 案例，在k8s安装部署jenkins
3.1安装nfs共享磁盘
-
所有node和磁盘共享服务器都安装nfs
[root@jenkins-master ~]# yum install nfs-utils -y [root@k8s-node1 ~]# yum install nfs-utils -y [root@k8s-node2 ~]# yum install nfs-utils -y [root@k8s-node3 ~]# yum install nfs-utils -y -
配置共享磁盘
[root@jenkins-master ~]# vim /etc/exports [root@jenkins-master ~]# cat /etc/exports /ifs/kubernetes *(rw,no_root_squash) -
创建共享磁盘目录
[root@jenkins-master ~]# mkdir -p /ifs/kubernetes/ -
每个node和磁盘共享都启动服务和设置开机启动
# systemctl start nfs #启动nfs # systemctl enable nfs # 设置开机启动 Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
3.2 安装pvc动态扩容插件
3.2.1 下载插件
-
克隆代码
[root@k8s-master ~]# git clone https://github.com/kubernetes-incubator/external-storage -
我这里是先下在了安装包，通过sftp上传
[root@k8s-master ~]# ll 总用量 212 -rw-------. 1 root root 1857 12月 15 09:32 anaconda-ks.cfg -rw-r--r-- 1 root root 187757 12月 15 12:54 calico.yaml -rw-r--r-- 1 root root 7591 12月 15 13:13 dashboard.yaml -rw-r--r-- 1 root root 6615 12月 16 20:17 ingress_controller.yaml drwxr-xr-x 2 root root 64 12月 30 16:33 nfs-client -rw-r--r--. 1 root root 7867 12月 15 10:22 open.sh drw-r--r-- 12 root root 144 12月 30 09:20 yaml
3.2.2 配置文件修改
-
进入插件里
[root@k8s-master ~]# cd nfs-client/ [root@k8s-master nfs-client]# ll 总用量 12 -rw-r--r-- 1 root root 225 12月 30 16:33 class.yaml # 创建存储类 -rw-r--r-- 1 root root 994 12月 30 16:33 deployment.yaml # 部署插件，需修改里面NFS服务器地址与共享目录 -rw-r--r-- 1 root root 1526 12月 30 16:33 rbac.yaml # 授权访问apiserver -
修改deployment配置文件
[root@k8s-master nfs-client]# vim deployment.yaml [root@k8s-master nfs-client]# cat deployment.yaml apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner --- kind: Deployment apiVersion: apps/v1 metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner # 把国外源修改为国内源 #image: quay.io/external_storage/nfs-client-provisioner:latest image: shichao01/nfs-client-provisioner:latest volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER # 修改nfs连接地址 #value: 192.168.1.61 value: 10.100.24.85 - name: NFS_PATH value: /ifs/kubernetes #共享路径，根据情况修改 volumes: - name: nfs-client-root nfs: server: 10.100.24.85 path: /ifs/kubernetes -
修改class文件
[root@k8s-master nfs-client]# vim class.yaml [root@k8s-master nfs-client]# cat class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "true" -
配置rbac.yaml文件
[root@k8s-master nfs-client]# vim rbac.yaml [root@k8s-master nfs-client]# cat rbac.yaml kind: ServiceAccount apiVersion: v1 metadata: name: nfs-client-provisioner --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io
3.2.3 启动所有服务
[root@k8s-master nfs-client]# kubectl apply -f .
storageclass.storage.k8s.io/managed-nfs-storage created
serviceaccount/nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
serviceaccount/nfs-client-provisioner unchanged
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
3.2.4 查看服务是否启动
[root@k8s-master nfs-client]# kubectl get pods
NAME READY STATUS RESTARTS AGE
configmap-demo-pod 1/1 Running 0 4d1h
my-hostpath 1/1 Running 4 2d1h
my-pod 1/1 Running 0 7h35m
nfs-client-provisioner-95c9579-m2f69 1/1 Running 0 52s
secret-demo-pod 1/1 Running 0 3d18h
web-nfs-84f8d7bf8d-6mj75 1/1 Running 0 44h
web-nfs-84f8d7bf8d-n4tpk 1/1 Running 0 44h
web-nfs-84f8d7bf8d-qvd2z 1/1 Running 0 44h
3.2.5 验证服务
[root@k8s-master nfs-client]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 3m46s
3.3 部署jenkins
-
创建jenkins的yaml配置文件路径
[root@k8s-master yaml]# mkdir -p jenkins-master [root@k8s-master yaml]# cd jenkins-master/ -
创建yaml的配置文件
-
编写statefulSet的jenkins配置文件
[root@k8s-master jenkins-master]# vim jenkins-statefulset.yaml [root@k8s-master jenkins-master]# cat jenkins-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: jenkins labels: name: jenkins spec: serviceName: "jenkins" replicas: 1 updateStrategy: type: RollingUpdate selector: matchLabels: name: jenkins template: metadata: name: jenkins labels: name: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts-alpine imagePullPolicy: Always ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 1 memory: 1Gi requests: cpu: 0.5 memory: 500Mi env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 securityContext: fsGroup: 1000 volumeClaimTemplates: - metadata: name: jenkins-home spec: storageClassName: "managed-nfs-storage" accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi -
编写service服务配置文件
[root@k8s-master jenkins-master]# vim jenkins-service.yaml [root@k8s-master jenkins-master]# cat jenkins-service.yaml apiVersion: v1 kind: Service metadata: name: jenkins namespace: default spec: selector: name: jenkins type: NodePort ports: - name: http port: 80 targetPort: 8080 protocol: TCP nodePort: 30006 - name: agent port: 50000 protocol: TCP -
编写ingress对外服务提供配置文件
[root@k8s-master jenkins-master]# vi jenkins-ingress.yaml [root@k8s-master jenkins-master]# cat jenkins-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: jenkins annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/tls-acme: "true" # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.ingress.kubernetes.io/proxy-request-buffering: "off" # nginx-ingress controller版本小于 0.9.0.beta-18 的配置 ingress.kubernetes.io/ssl-redirect: "true" ingress.kubernetes.io/proxy-body-size: 50m ingress.kubernetes.io/proxy-request-buffering: "off" spec: rules: - host: jenkins.scajy.org http: paths: - path: / backend: serviceName: jenkins servicePort: 80 -
配置jenkins的rbac可以访问api资源
[root@k8s-master jenkins-master]# vim jenkins-rbac.yaml [root@k8s-master jenkins-master]# cat jenkins-rbac.yaml --- # 创建名为jenkins的ServiceAccount apiVersion: v1 kind: ServiceAccount metadata: name: jenkins --- # 创建名为jenkins的Role，授予允许管理API组的资源Pod kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- # 将名为jenkins的Role绑定到名为jenkins的ServiceAccount apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins -
配置jenkins的account的rbac权限
[root@k8s-master jenkins-master]# vim jenkins-account.yaml [root@k8s-master jenkins-master]# cat jenkins-account.yaml # In GKE need to get RBAC permissions first with # kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>] --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins -
启动所有配置文件
[root@k8s-master jenkins]# kubectl apply -f . Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress ingress.extensions/jenkins created serviceaccount/jenkins created Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role role.rbac.authorization.k8s.io/jenkins created Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding rolebinding.rbac.authorization.k8s.io/jenkins created serviceaccount/jenkins unchanged role.rbac.authorization.k8s.io/jenkins unchanged rolebinding.rbac.authorization.k8s.io/jenkins unchanged service/jenkins created statefulset.apps/jenkins created
-
3.4 检查服务是否启动
[root@k8s-master jenkins-master]# kubectl get pods,service -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/jenkins-0 1/1 Running 8 127m 10.244.36.72 k8s-node1 <none> <none>
pod/nfs-client-provisioner-5d99f8d7b9-562ff 1/1 Running 1 12d 10.244.169.130 k8s-node2 <none> <none>
pod/nginx-6799fc88d8-9fchc 1/1 Running 0 12d 10.244.169.131 k8s-node2 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/jenkins NodePort 10.107.141.118 <none> 80:30006/TCP,50000:32291/TCP 163m name=jenkins
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 13d <none>
service/nginx NodePort 10.99.236.10 <none> 80:30939/TCP 12d app=nginx
[root@k8s-master jenkins-master]# kubectl get ep
NAME ENDPOINTS AGE
fuseim.pri-ifs <none> 12d
jenkins 10.244.36.72:8080,10.244.36.72:50000 164m
kubernetes 10.100.24.81:6443 13d
nginx 10.244.169.131:80 12d
3.5 使用浏览器访问测试
-
这里先绑定ingress的hosts地址
-
使用域名访问测试
-
使用IP访问测试
-
注释：
- 到这里，我们发现都可以访问
3.6 浏览器访问进行jenkins安装
-
使用命令查看密码
[root@k8s-master jenkins-master]# kubectl exec -it jenkins-0 cat /var/jenkins_home/secrets/initialAdminPassword kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. 8848ef969b454f849121e5790ea7dd52
- 跳过插件安装
