deploy jenkins by kubernetes

kubernetes

https://kubernetes.io/

K8S是自动化部署，伸缩，容器应用的管理工具。

Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.

https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

优点：

服务发现

负责均衡

存储调度

自动上线和回滚

自愈

密钥和配置管理

Kubernetes provides you with:

• Service discovery and load balancing Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
• Storage orchestration Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.
• Automated rollouts and rollbacks You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.
• Automatic bin packing You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.
• Self-healing Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.
• Secret and configuration management Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.

Jenkins by K8S

https://www.jenkins.io/doc/book/installing/kubernetes/#kubernetes

通过编排容器部署，有效控制资源的数量。

自动部署，伸缩，容器管理。

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

A Kubernetes cluster adds a new automation layer to Jenkins. Kubernetes makes sure that resources are used effectively and that your servers and underlying infrastructure are not overloaded. Kubernetes' ability to orchestrate container deployment ensures that Jenkins always has the right amount of resources available. This section will describe different options to install/run Jenkins on Kubernetes.

How to Install Jenkins on Kubernetes Cluster

https://phoenixnap.com/kb/how-to-install-jenkins-kubernetes

https://computingforgeeks.com/separating-jenkinsfile-and-deployment-yaml-manifests-from-developers-gitlab-sources/

https://devopscube.com/setup-jenkins-on-kubernetes-cluster/

jenkins-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        ports:
        - containerPort: 8080
        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8081
      targetPort: 8080
      nodePort: 32767
  selector:
    app: jenkins

nodePort service

https://www.ithands-on.com/2021/10/kubernetes-101-nodeport-service-example.html

使得POD中的服务可以被外界访问。

通过 NODE 的 IP 和 端口。

A nodePort service makes the application running inside the pods accessible from the outside on the same port on all the nodes of the cluster.

The nodePort service has access to all the nodes of the cluster.

 

 

如下执行过程。

Below is a Yaml file of a nodePort service:

 

 

• TargetPort: it exposes the application running inside the pod. The nodePort service forwards the requests to that port.
• Port: the port on which the service is accessed.
• NodePort: the open port on the node.

 

POD绑定具体NODE

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

在service中，通过 nodeSelector 来寻找对应 node，可以告诉用户使用 node ip 和 端口访问。

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    disktype: ssd

SESSIONAFFINITY + COOKIE

https://stackoverflow.com/questions/56323438/session-affinity-settings-for-multiple-pods-exposed-by-a-single-service

 

Main concept of Session Affinity is to redirect traffic from one client always to specific node. Please keep in mind that session affinity is a best-effort method and there are scenarios where it will fail due to pod restarts or network errors. There are two main types of Session Affinity:

1) Based on Client IP

This option works well for scenario where there is only one client per IP. In this method you don't need Ingress/Proxy between K8s services and client. Client IP should be static, because each time when client will change IP he will be redirected to another pod.

To enable the session affinity in kubernetes, we can add the following to the service definition.

service.spec.sessionAffinity: ClientIP

Because community provided proper manifest to use this method I will not duplicate.

2) Based on Cookies

It works when there are multiple clients from the same IP, because it´s stored at web browser level. This method require Ingress object. Steps to apply this method with more detailed information can be found here under Session affinity based on Cookie section.

• Create NGINX controller deployment
• Create NGINX service
• Create Ingress
• Redirect your public DNS name to the NGINX service public/external IP.

About mapping ClientIP and POD, according to Documentation kube-proxy is responsible for SessionAffinity. One of Kube-Proxy job is writing to IPtables, more details here so thats how it is mapped.

完美状态

https://linuxtech.cn/article/167/

jenkins本身使用k8s部署，同时jenkins的agents使用k8s集群功能实现。

持续构建与发布是我们日常工作中必不可少的一个步骤，目前大多公司都采用 Jenkins 集群来搭建符合需求的 CI/CD 流程，然而传统的 Jenkins Slave 一主多从方式会存在一些痛点，比如：
主 Master 发生单点故障时，整个流程都不可用了；
每个Slave的配置环境不一样，来完成不同语言的编译打包等操作，但是这些差异化的配置导致管理起来非常不方便，维护起来也是比较费劲；
资源分配不均衡，有的 Slave 要运行的 job 出现排队等待，而有的 Slave 处于空闲状态；
最后资源有浪费，每台 Slave 可能是实体机或者 VM，当 Slave 处于空闲状态时，也不会完全释放掉资源。

Jenkins Master 和 Jenkins Slave 以 Docker Container 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。

这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Docker Container 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且 Docker Container 也会自动删除，恢复到最初状态。

这种方式带来的好处有很多：
服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。
动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。