kubernetes

认识kubernetes

https://github.com/gjmzj/kubeasz

service是核心，service是由pod组成的，pod是由容器组成的，提供service的是容器，service和pod通过标签关联，pod运行在Node上，每个pod都有一个特殊的容器叫pause（共享网络、共享数据），其他容器叫做业务容器，

https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/k8s

1.是一个开源的，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效（powerful）,Kubernetes提供了应用部署，规划，更新，维护的一种机制。

http://docs.kubernetes.org.cn/227.html

https://www.cnblogs.com/xhyan/p/6656062.html

https://www.cnblogs.com/fengjian2016/p/6392900.html

https://kubernetes.io/zh/docs/tutorials/kubernetes-basics/

2.安装kubernetes，关闭防火墙

[root@centos-01 ~]# systemctl stop firewalld
[root@centos-01 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@centos-01 ~]# setenforce 0
setenforce: SELinux is disabled
[root@centos-01 ~]# 

3.安装etcd（作用存储kubernetes里面的配置文件）和kubernetes

[root@centos-01 ~]# yum install -y etcd kubernetes

4.修改配置文件，将--selinux-enabled 改为 --selinux-enabled=false  --insecure-registry gcr.io

[root@centos-01 ~]# vim /etc/sysconfig/docker

OPTIONS='--selinux-enabled=false --insecure-registry gcr.io --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

5.编辑apiserver配置文件，把--admission_control参数中的ServiceAccount删除

[root@centos-01 ~]# vim /etc/kubernetes/apiserver 

6.准备工作，安装python-rhsm-certificates包，如果提示python-rhsm-certificates-1.19.10-1.el7_4.x86_64 被已安装的 subscription-manager-rhsm-certificates1.20.11-1.el7.centos.x86_64 取代

yum install python-rhsm-certificates

　　

[root@centos-01 ~]# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
--2018-12-11 04:01:39--  http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
正在解析主机 mirror.centos.org (mirror.centos.org)... 213.184.126.230, 2605:9000:401:102::2
正在连接 mirror.centos.org (mirror.centos.org)|213.184.126.230|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：42188 (41K) [application/x-rpm]
正在保存至: “python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm”

100%[======================================================================================>] 42,188      66.1KB/s 用时 0.6s   

2018-12-11 04:01:40 (66.1 KB/s) - 已保存 “python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm” [42188/42188])

[root@centos-01 ~]# rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm |cpio -iv --to-stdout ./etc/rhsm/ca/redhatuep.pem > /etc/rhsm/ca/redhat-uep.pem
17 块
[root@centos-01 ~]# 

7.配置docker加速器

vi /etc/docker/daemon.json//加入如下内容
{
 "registry-mirrors": ["https://dhq9bx4f.mirror.aliyuncs.com"]
}

8.按顺序启动所有服务（红的是master节点上的，绿的是）

for s in etcd docker kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
do
 systemctl start $s
done

9.创建一个rc文件

vim mysql-rc.yaml

apiVersion: v1
kind: ReplicationController #副本控制器RC
metadata:
  name: mysql #RC的名称，全局唯一
spec:
  replicas: 1 #Pod副本的期待数量
  selector:
    app: mysql #符合目标的Pod拥有此标签
  template: #根据此模板创建Pod的副本（实例）
    metadata:
      labels:
        app: mysql #Pod副本拥有的标签，对应RC的Selector
    spec:
      containers: #Pod内容器的定义部分
      - name: mysql #容器的名称
        image: mysql:5.6 #容器对应的Docker image
        ports:
        - containerPort: 3306 #容器应用监听的端口号
        env: #注入容器内的环境变量
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"

10.创建rc

[root@centos-01 ~]# kubectl create -f mysql-rc.yaml 
replicationcontroller "mysql" created

查看是否pull成功了镜像，如果没有pull成功需要手动pull
docker images

docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest
docker pull mysql:5.6

　　

11.查看命令干了什么

[root@centos-01 ~]# tail /var/log/messages（其实是docker在下载mysql镜像）

12.查看都有哪些rc

[root@centos-01 ~]# kubectl get rc
NAME      DESIRED   CURRENT   READY     AGE
mysql     1         1         0         5m

13.查看pod状态（状态变成running说明没问题）

[root@centos-01 ~]# kubectl get pod
NAME          READY     STATUS    RESTARTS   AGE
mysql-b57jv   0/1       Pending   0          7m
[root@centos-01 ~]#

[root@centos-02 rhsm]# kubectl get pod       
NAME          READY     STATUS    RESTARTS   AGE
mysql-n1jtc   1/1       Running   0          21m
[root@centos-02 rhsm]# 

14.查看service

[root@centos-01 ~]# kubectl get service

15.创建service（svc）文件

[root@centos-02 ~]# vim  mysql-svc.yaml

　　

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql

[root@centos-02 ~]# kubectl create -f mysql-svc.yaml
service "mysql" created
[root@centos-02 ~]# 
[root@centos-02 ~]# kubectl get svc
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes   10.254.0.1      <none>        443/TCP    35m
mysql        10.254.73.183（就是我们的serviceIP）   <none>        3306/TCP   35s
[root@centos-02 ~]# 

16.安装mysql

[root@centos-02 ~]# yum install -y mysql

17.这样我们就可以通过10.254.73.183：3306访问mysql了

[root@centos-02 ~]# mysql -uroot -p123456 -h10.254.73.183
Welcome to the MariaDB monitor.  Commands end with ; or g.
Your MySQL connection id is 1
Server version: 5.6.42 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MySQL [(none)]> 

18.创建myweb

[root@centos-02 ~]# vim myweb-rc.yaml
kind: ReplicationController
metadata:
  name: myweb
spec:
  replicas: 1
  selector:
    app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
        - name: myweb
          image: kubeguide/tomcat-app:v1
          ports:
          - containerPort: 8080
          env:
          - name: MYSQL_SERVICE_HOST
            value: '10.254.73.183' #这里的IP需要通过kubect get svc 查看mysql的cluster ip（10.254.73.183）
          - name: MYSQL_SERVICE_PORT
            value: '3306'

[root@centos-02 ~]# kubectl create -f myweb-rc.yaml 
replicationcontroller "myweb" created
[root@centos-02 ~]# 

19.查看pod

[root@centos-02 ~]# kubectl get pod
NAME          READY     STATUS              RESTARTS   AGE
mysql-n1jtc   1/1       Running             0          2h
myweb-1x5h9   0/1       ContainerCreating   0          1m
[root@centos-02 ~]# docker images
REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
docker.io/mysql                                       5.6                 a876cc5d29e4        3 weeks ago         256 MB
registry.access.redhat.com/rhel7/pod-infrastructure   latest              99965fb98423        14 months ago       209 MB
[root@centos-02 ~]# 

20.创建service

[root@centos-02 ~]# vim myweb-svc.yaml
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30001
  selector:
    app: myweb

[root@centos-02 ~]# kubectl create -f myweb-svc.yaml 
service "myweb" created
[root@centos-02 ~]# 

21.查看pod和service

[root@centos-02 ~]# kubectl get pod
NAME          READY     STATUS    RESTARTS   AGE
mysql-n1jtc   1/1       Running   0          2h
myweb-1x5h9   1/1       Running   0          8m
[root@centos-02 ~]# kubectl get svc
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   10.254.0.1      <none>        443/TCP          2h
mysql        10.254.73.183   <none>        3306/TCP         2h
myweb        10.254.51.166   <nodes>       8080:30001/TCP   1m
[root@centos-02 ~]# 

22.访问tomcat

[root@centos-02 ~]# curl -I 10.254.51.166:8080
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 13 Dec 2018 20:01:56 GMT

[root@centos-02 ~]# 

[root@centos-02 ~]# curl 10.254.51.166:8080/demo/

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>HPE University Docker&Kubernetes Learning</title>
</head>
<body  align="center">


      <h2>Congratulations!!</h2>
     <br></br>
         <input type="button" value="Add..." onclick="location.href='input.html'" >
             <br></br>
      <TABLE align="center"  border="1" width="600px">
   <TR>
      <TD>Name</TD>
      <TD>Level(Score)</TD>
   </TR>

      
 <TR>
      <TD>google</TD>
      <TD>100</TD>
   </TR>

 <TR>
      <TD>docker</TD>
      <TD>100</TD>
   </TR>

 <TR>
      <TD>teacher</TD>
      <TD>100</TD>
   </TR>

 <TR>
      <TD>HPE</TD>
      <TD>100</TD>
   </TR>

 <TR>
      <TD>our team</TD>
      <TD>100</TD>
   </TR>

 <TR>
      <TD>me</TD>
      <TD>100</TD>
   </TR>

  </TABLE>
      
</body>
</html>
[root@centos-02 ~]# 

[root@centos-02 ~]# curl 192.168.242.132:30001/demo/

23.通过浏览器访问，我们发现默认FORWARD是DROP,我们需要不FORWARD打开

[root@centos-02 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 4 packets, 248 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 537K  532M KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   96 11569 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   96 11569 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
   78 10629 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    3   180 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 3 packets, 156 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 507K  188M KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 508K  188M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   96 11569 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
[root@centos-02 ~]# 

[root@centos-02 ~]# iptables -P FORWARD ACCEPT
[root@centos-02 ~]# 

24.成功访问

25.我们发现多了一个HPE_APP表

[root@centos-02 ~]# mysql -uroot -p123456 -h10.254.73.183     
Welcome to the MariaDB monitor.  Commands end with ; or g.
Your MySQL connection id is 11
Server version: 5.6.42 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| HPE_APP            |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)

MySQL [(none)]> 

　

MySQL [(none)]> use HPE_APP;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MySQL [HPE_APP]> SHOW TABLES;
+-------------------+
| Tables_in_HPE_APP |
+-------------------+
| T_USERS           |
+-------------------+
1 row in set (0.00 sec)

MySQL [HPE_APP]> SELECT * FROM T_USERS;
+----+-------------+-------+
| ID | USER_NAME   | LEVEL |
+----+-------------+-------+
|  1 | me          | 100   |
|  2 | our team    | 100   |
|  3 | HPE         | 100   |
|  4 | teacher     | 100   |
|  5 | docker      | 100   |
|  6 | google      | 100   |
|  7 | 15001316083 | 100   |
+----+-------------+-------+
7 rows in set (0.00 sec)

MySQL [HPE_APP]> 

26.命令总结

[root@centos-02 ~]# kubectl create -f ^C
[root@centos-02 ~]# kubectl get pod
NAME          READY     STATUS    RESTARTS   AGE
mysql-n1jtc   1/1       Running   0          3h
myweb-1x5h9   1/1       Running   0          25m
[root@centos-02 ~]# kubectl get rc
NAME      DESIRED   CURRENT   READY     AGE
mysql     1         1         1         3h
myweb     1         1         1         25m
[root@centos-02 ~]# kubectl get svc
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   10.254.0.1      <none>        443/TCP          3h
mysql        10.254.73.183   <none>        3306/TCP         2h
myweb        10.254.51.166   <nodes>       8080:30001/TCP   18m
[root@centos-02 ~]# 

kubernetes相关概念　　

1.kubernetes从物理上划分为master节点和node节点

2.RC中动态修改pod副本数量，下面两个rc分别有一个动态的pod，我们动态调整成2个mysql pod

[root@centos-02 ~]# kubectl get rc
NAME      DESIRED   CURRENT   READY     AGE
mysql     1         1         1         21h
myweb     1         1         1         19h
[root@centos-02 ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
mysql-n1jtc   1/1       Running   0          21h
myweb-1x5h9   1/1       Running   0          19h
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl scale rc mysql --replicas=2    
replicationcontroller "mysql" scaled
[root@centos-02 ~]# kubectl get pods
NAME          READY     STATUS              RESTARTS   AGE
mysql-cc1tx   0/1       ContainerCreating   0          6s
mysql-n1jtc   1/1       Running             0          21h
myweb-1x5h9   1/1       Running             0          19h
[root@centos-02 ~]# kubectl get rc  
NAME      DESIRED   CURRENT   READY     AGE
mysql     2         2         2         21h
myweb     1         1         1         19h
[root@centos-02 ~]# 

3.删除RC,RC对应的pod也会被删除掉

[root@centos-02 ~]# kubectl get rc
NAME      DESIRED   CURRENT   READY     AGE
mysql     2         2         2         22h
myweb     1         1         1         19h
[root@centos-02 ~]# kubectl delete rc myweb
replicationcontroller "myweb" deleted
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl get rc
NAME      DESIRED   CURRENT   READY     AGE
mysql     2         2         2         22h
myweb     1         1         1         19h
[root@centos-02 ~]# kubectl delete rc myweb
replicationcontroller "myweb" deleted
[root@centos-02 ~]# kubectl get rc         
NAME      DESIRED   CURRENT   READY     AGE
mysql     2         2         2         22h
[root@centos-02 ~]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
mysql-cc1tx   1/1       Running   0          14m
mysql-n1jtc   1/1       Running   0          22h
[root@centos-02 ~]# 

4.svc中还是有myweb，需要手动删掉

[root@centos-02 ~]# kubectl get svc
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   10.254.0.1      <none>        443/TCP          22h
mysql        10.254.73.183   <none>        3306/TCP         21h
myweb        10.254.51.166   <nodes>       8080:30001/TCP   19h
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl delete svc myweb
service "myweb" deleted
[root@centos-02 ~]# kubectl get svc         
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes   10.254.0.1      <none>        443/TCP    22h
mysql        10.254.73.183   <none>        3306/TCP   21h
[root@centos-02 ~]# 

5.Deployment 在1.2版本引入的概念，目的是为了解决pod编排问题，在内部使用了Replica Set，它和RC比较，相似度为90%以上，可以认为 是RC的升级版。 跟RC比较，最大的一个特点是可以知道pod部署的进度。　

　Deployment示例：

[root@centos-02 ~]# vim fr-dp.yaml
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: frontend
    matchExpressions:
      - {key: tier, operator: In, values: [frontend]}
  template:
    metadata:
      labels:
        app: app-demo
        tier: frontend
    spec:
      containers:
      - name: tomcat-demo
        image: tomcat
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

6.创建frontend

[root@centos-02 ~]# kubectl create -f fr-dp.yaml
deployment "frontend" created
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl get pods
NAME                       READY     STATUS              RESTARTS   AGE
frontend-141477217-20031   0/1       ContainerCreating   0          45s
mysql-cc1tx                1/1       Running             0          1h
mysql-n1jtc                1/1       Running             0          23h
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl get deployment
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
frontend   1         1         1            0           1m
[root@centos-02 ~]# 

7.查看pod情况

[root@centos-02 ~]# kubectl describe pod frontend-141477217-20031
Name:           frontend-141477217-20031
Namespace:      default
Node:           127.0.0.1/127.0.0.1
Start Time:     Sat, 15 Dec 2018 00:15:50 +0800
Labels:         app=app-demo
                pod-template-hash=141477217
                tier=frontend
Status:         Pending
IP:
Controllers:    ReplicaSet/frontend-141477217
Containers:
  tomcat-demo:
    Container ID:
    Image:                      tomcat
    Image ID:
    Port:                       8080/TCP
    State:                      Waiting
      Reason:                   ContainerCreating
    Ready:                      False
    Restart Count:              0
    Volume Mounts:              <none>
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True 
  Ready         False 
  PodScheduled  True 
No volumes.
QoS Class:      BestEffort
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath  Type             Reason                  Message
  ---------     --------        -----   ----                    -------------  -------- ------                  -------
  4m            4m              1       {default-scheduler }                   Normal           Scheduled               Successfully assigned frontend-141477217-20031 to 127.0.0.1
  4m            4m              1       {kubelet 127.0.0.1}                    Warning          MissingClusterDNS       kubelet does not have ClusterDNS IP configured and cannot create Pod 
using "ClusterFirst" policy. Falling back to DNSDefault policy.
  4m            4m              1       {kubelet 127.0.0.1}     spec.containers{tomcat-demo}    Normal          Pulling                 pulling image "tomcat"
[root@centos-02 ~]# 

8.查看下有没有pull下来tomcat的镜像　　

[root@centos-02 ~]# docker images
REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
docker.io/tomcat                                      latest              48dd385504b1        6 days ago          475 MB
docker.io/mysql                                       5.6                 a876cc5d29e4        4 weeks ago         256 MB
registry.access.redhat.com/rhel7/pod-infrastructure   latest              99965fb98423        14 months ago       209 MB
docker.io/kubeguide/tomcat-app                        v1                  a29e200a18e9        2 years ago         358 MB
[root@centos-02 ~]# 

9.HPA:在1.1版本，kubernetes官方发布了HPA，实现pod的动态扩容、缩容，它属于一种kubernetes的资源对象。它通过追踪分析 RC控制的所有目标pod的负载变化情况，来决定是否需要针对性地调整目标Pod的副本数，这是HPA的实现原理。

pod负载度量指标：
1）CpuUtilizationPercentage
目标pod所有副本自身的cpu利用率平用均值。一个pod自身的cpu利用率＝该pod当前cpu的使用量／pod Request值。如果某
一个时刻，CPUUtilizationPercentage的值超过了80%，则判定当前的pod已经不够支撑业务，需要增加pod。
2）应用程序自定义的度量指标，比如服务每秒内的请求数（TPS或QPS）
HPA示例：
apiVerion: autosacling/v1
kind: HorizontalPodAutoscaler
metadata:
 name: php-apache
 namespace: default
spec:
 maxReplicas: 10
 minReplicas: 1
 scaleTargetRef:
 kind: Deployment
 name: php-apache
 targetCPUUtilizationPercentage: 90
说明：HPA控制的目标对象是一个名叫php-apache的Deployment里的pod副本，当cpu平均值超过90%时就会扩容，pod副本
数控制范围是1－10.
除了以上的xml文件定义HPA外，也可以用命令行的方式来定义：
kubectl autoscale deployment php-apache --cpu-percent=90 --min=1 --max=10

10.Service是kubernetes中最核心的资源对象之一，Service可以理解成是微服务架构中的一个“微服务”，pod、RC、 Deployment都是为Service提供嫁衣的。

简单讲一个service本质上是一组pod组成的一个集群，前面我们说过service和pod之间是通过Label来串起来的，相同Service的 pod的Label一样。同一个service下的所有pod是通过kube-proxy实现负载均衡，而每个service都会分配一个全局唯一的虚拟 ip，也叫做cluster ip。在该service整个生命周期内，cluster ip是不会改变的，而在kubernetes中还有一个dns服务，它把 service的name和cluster ip映射起来。

11.查看pod的IP地址以及端口

[root@centos-02 ~]# kubectl get endpoints
NAME         ENDPOINTS                         AGE
kubernetes   192.168.242.132:6443              23h
mysql        172.17.0.2:3306,172.17.0.4:3306   23h
[root@centos-02 ~]# 

12.查看service分配的cluster ip

[root@centos-02 ~]# kubectl get svc mysql -o yaml               
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-12-13T17:43:37Z
  name: mysql
  namespace: default
  resourceVersion: "2329"
  selfLink: /api/v1/namespaces/default/services/mysql
  uid: 9ebfd5d8-fefe-11e8-b6e3-000c2959c2d2
spec:
  clusterIP: 10.254.73.183
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
[root@centos-02 ~]# 

13.Namespace当kubernetes集群中存在多租户的情况下，就需要有一种机制实现每个租户的资源隔离。而namespace的目的就是为了实现资 源隔离。

查看集群所有的namespace

[root@centos-02 ~]# kubectl get namespace
NAME          STATUS    AGE
default       Active    1d
kube-system   Active    1d
[root@centos-02 ~]# 

[root@centos-02 ~]# vim dev-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: dev

14.创建dev namespace

[root@centos-02 ~]# kubectl create -f dev-ns.yaml 
namespace "dev" created
[root@centos-02 ~]# 

15.获取namespace

[root@centos-02 ~]# kubectl get ns
NAME          STATUS    AGE
default       Active    1d
dev           Active    37s
kube-system   Active    1d
[root@centos-02 ~]# 

16.定义pod

[root@centos-02 ~]# vim busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: dev
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "500"
    name: busybox

[root@centos-02 ~]# kubectl create -f busybox-pod.yaml 
pod "busybox" created
[root@centos-02 ~]# 

17.我们直接get pods不能查看到busybox，需要指定namespace为dev查看

[root@centos-02 ~]# kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
frontend-141477217-20031   1/1       Running   0          1h
mysql-cc1tx                1/1       Running   0          3h
mysql-n1jtc                1/1       Running   0          1d
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl get pods -n dev
NAME      READY     STATUS    RESTARTS   AGE
busybox   1/1       Running   0          4m
[root@centos-02 ~]# 

[root@centos-02 ~]# kubectl get pods --all-namespaces
NAMESPACE   NAME                       READY     STATUS    RESTARTS   AGE
default     frontend-141477217-20031   1/1       Running   0          1h
default     mysql-cc1tx                1/1       Running   0          3h
default     mysql-n1jtc                1/1       Running   0          1d
dev         busybox                    1/1       Running   0          5m
[root@centos-02 ~]# 

kubectl get pods -n dev

kubectl命令用法

语法：
kubectl [command] [TYPE] [NAME] [flags]
1 command：子命令，用于操作Kubernetes集群资源对象的命令，如create, delete, describe, get, apply等
2 TYPE：资源对象的类型，如pod, service, rc, deployment, node等，可以单数、复数以及简写（pod, pods, po/service,
services, svc）
3 NAME：资源对象的名称，不指定则返回所有，如get pod 会返回所有pod， get pod nginx， 只返回nginx这个pod
4 flags：kubectl子命令的可选参数，例如-n 指定namespace，-s 指定apiserver的URL

资源对象类型列表
可以用这个命令获取到：
kubectl explain
或
kubectl api-resources

名称 简写
componentsstatuses cs
daemonsets ds
deployment deploy
events ev
endpoints ep
horizontalpodautoscalers hpa
ingresses ing
jobs
limitranges limits
nodes no
namspaces ns
pods po
persistentvolumes pv
persistentvolumeclaims pvc
resourcequotas quota
replicationcontrollers rc
secrets
serviceaccounts sa
services svc

特殊用法：
kubectl get pods pod1 pod2
kubectl get pod/pod1 rc/rc1
kubectl create -f pod1.yaml -f rc1.yaml -f service1.yaml

kubectl子命令
主要包括对资源的创建、删除、查看、修改、配置、运行等
kubectl --help 可以查看所有子命令
kubectl参数
kubectl options 可以查看支持的参数，例如--namespace指定所在namespace
kubectl输出格式
kubectl命令可以用多种格式对结果进行显示，输出格式通过-o参数指定：
-o支持的格式有
输出格式 说明
custom-columns=<spec> 根据自定义列名进行输出，逗号分隔
custom-columns-file=<filename> 从文件中获取自定义列名进行输出
json 以JSON格式显示结果
jsonpath=<template> 输出jasonpath表达式定义的字段信息
jasonpath-file=<filename> 输出jsonpath表达式定义的字段信息，来源于文件
name 仅输出资源对象的名称
wide 输出更多信息，比如会输出node名
yaml 以yaml格式输出
举例：
kubectl get pod -o wide
kubectl get pod -o yaml
kubectl get pod -o custom-columns=NAME:.metadata.name,RESC:.metadata.resourceVersion
kubectl get pod --sort-by=.metadata.name //按name排序
kubectl命令示例：
1）创建资源对象
根据yaml文件创建service和deployment
kubectl create -f my-service.yaml -f my-deploy.yaml
也可以指定一个目录，这样可以一次性根据该目录下所有yaml或json文件定义资源
kubectl create -f <directory>
2）查看资源对象
查看所有pod
kubectl get pods
查看deployment和service
kubectl get deploy,svc
3）描述资源对象
显示node的详细信息
kubectl describe nodes <node-name>
显示pod的详细信息
kubectl describe pods/<pod-name>
显示deployment管理的pod信息
kubectl describe pods <deployment-name>
4）删除资源对象
基于yaml文件删除
kubectl delete -f pod.yaml
删除所有包含某个label的pod和service
kubectl delete po,svc -l name=<lable-name>
删除所有pod
kubectl delete po --all
5）执行容器的命令
在pod中执行某个命令，如date
kubectl exec <pod-name> date //pod-name如果不加，默认会选择第一个pod
指定pod的某个容器执行命令
kubectl exec <pod-name> date
进入到pod的容器里
kubectl exec -it <pod-name> bash
6）查看容器日志
kubectl logs <pod-name>
可以动态查看，类似于tail -f
kubectl logs -f <pod-name> -c <container-name>

 搭建kubernetes集群（ansible-playbook）-1

1.软硬件限制（详情见https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/k8s）

cpu和内存 master：至少1核两g，推荐两核4g，node至少1核2g

linux系统内核版本至少3.10，推荐centos7/RHEL7

docker 至少1.9版本，推荐1.12+

etcd至少2.0版本，推荐3.0+

2.四台机器全部执行

yum update
yum install epel-release
yum install python

3.deploy节点安装和准备ansible

(1)130服务器安装pip

yum install -y python-pip git　　

 (2)升级pip源

pip install pip --upgrade -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

(3)安装ansible （pip和yum挺像的主要用于安装python下的插件），如果这种方式安装失败用yum安装（yum list|grep ansible、 yum install -y ansible）

[root@centos-04 ~]# pip install --no-cache-dir ansible -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

Installing collected packages: MarkupSafe, jinja2, PyYAML, idna, enum34, six, pycparser, cffi, asn1crypto, cryptography, pynacl, pyasn1, bcrypt, paramiko, ansible
  Running setup.py install for PyYAML ... done
  Running setup.py install for pycparser ... done
  Running setup.py install for ansible ... done
Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.5 asn1crypto-0.24.0 bcrypt-3.1.5 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 jinja2-2.10 paramiko-2.4.2 pyasn1-0.4.4 
pycparser-2.19 pynacl-1.3.0 six-1.12.0
[root@centos-04 ~]# 

deploy节点配置免密码登录

1.生成密钥对

[root@centos-04 ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):             
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:qrghr27RSPWCV5mBazMJiT6V3KDX0+s9twBLSnjemac root@centos-04
The key's randomart image is:
+---[RSA 2048]----+
|. o.=.+          |
|.o.*.=.          |
|..=.=o .         |
| =.O... .        |
|. *.+o +S        |
| o .+ =.*        |
|. o  o.* = .     |
| + o .  o + .    |
|+o+..  E   .     |
+----[SHA256]-----+
[root@centos-04 ~]# 

[root@centos-04 ~]# for ip in 130 131 132 133; do ssh-copy-id 192.168.242.$ip; done

2.登录各个机器测试（ctrl+d退出）

[root@centos-04 ~]# for ip in 130 131 132 133; do ssh 192.168.242.$ip; done        
Last login: Tue Dec 18 19:04:47 2018 from 192.168.242.1
[root@centos-04 ~]# 登出
Connection to 192.168.242.130 closed.
Last login: Tue Dec 18 19:08:23 2018 from 192.168.242.1
ABRT 已检测到 '4' 个问题。预了解详细信息请执行：abrt-cli list --since 1545131303
[root@centos-01 ~]# 登出
Connection to 192.168.242.131 closed.
Last login: Tue Dec 18 19:08:14 2018 from 192.168.242.1
[root@centos-02 ~]# 登出
Connection to 192.168.242.132 closed.
Last login: Tue Dec 18 19:06:44 2018 from 192.168.242.1
[root@centos-03 ~]# 登出
Connection to 192.168.242.133 closed.
[root@centos-04 ~]# 

deploy上编排k8s

[root@centos-04 ~]# git clone https://github.com/gjmzj/kubeasz.git
[root@centos-04 ~]# mkdir -p /etc/ansible
[root@centos-04 ~]# mv kubeasz/* /etc/ansible/

[root@centos-04 ~]# cd /etc/ansible/
[root@centos-04 ansible]# du -sh
2.6M    .
[root@centos-04 ansible]# ls
01.prepare.yml      05.kube-node.yml      20.addnode.yml    24.restore.yml  bin      manifests  tools
02.etcd.yml         06.network.yml        21.addmaster.yml  90.setup.yml    docs     pics
03.docker.yml       07.cluster-addon.yml  22.upgrade.yml    99.clean.yml    down     README.md
04.kube-master.yml  11.harbor.yml         23.backup.yml     ansible.cfg     example  roles
[root@centos-04 ansible]# 　　

配置集群参数

[root@centos-04 ansible]# cp example/hosts.m-masters.example hosts
[root@centos-04 ansible]# 

[root@centos-04 ansible]# vim hosts （根据实际情况修改IP地址）
[deploy]
192.168.242.130 NTP_ENABLED=no

[etcd]
192.168.242.130 NODE_NAME=etcd1
192.168.242.131 NODE_NAME=etcd2
192.168.242.132 NODE_NAME=etcd3

[kube-master]
192.168.242.130
192.168.242.133

[lb]
192.168.242.130 LB_IF="ens33" LB_ROLE=backup
192.168.242.133 LB_IF="eno16777736"  LB_ROLE=master

[kube-node]
192.168.242.131
192.168.242.132

K8S_VER="v1.11"

MASTER_IP="192.168.242.150"　　

从百度云网盘下载二进制文件 https://pan.baidu.com/s/1c4RFaA#list/path=%2F 可以根据自己所需版本，下载对应的tar包，这里我下载1.11 经过一番折腾，最终把k8s.1-11-2.tar.gz的tar包放到了depoly上，上传包-解压-移动到bin目录

[root@centos-04 ~]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
  100%  214046 KB 9306 KB/s 00:00:23       0 Errorss

[root@centos-04 ~]# ls
anaconda-ks.cfg  k8s.1-11-3.tar.gz  kubeasz
[root@centos-04 ~]# 

tar zxvf k8s.1-11-2.tar.gz
mv bin/* /etc/ansible/bin/

[root@centos-04 ~]# cd /etc/ansible/bin/
[root@centos-04 bin]# ls
bridge          docker                  dockerd       etcdctl         kube-controller-manager  loopback
calicoctl       docker-compose          docker-init   flannel         kubectl                  portmap
cfssl           docker-containerd       docker-proxy  helm            kubelet                  readme.md
cfssl-certinfo  docker-containerd-ctr   docker-runc   host-local      kube-proxy
cfssljson       docker-containerd-shim  etcd          kube-apiserver  kube-scheduler
[root@centos-04 bin]# 

创建证书和安装准备

[root@centos-04 ansible]# ansible-playbook 01.prepare.yml

安装etcd集群

[root@centos-04 ansible]# ansible-playbook 02.etcd.yml

检查etcd节点健康状况：（如果提示etcdctl命令不存在，先执行bash）

for ip in 130 131 132 ; do ETCDCTL_API=3 etcdctl --endpoints=https://192.168.242.$ip:2379 --
cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done

安装docker

ansible-playbook 03.docker.yml

安装master节点

ansible-playbook 04.kube-master.yml

查看集群状态

kubectl get componentstatus 

安装node节点

[root@centos-04 ansible]# ansible-playbook 05.kube-node.yml

查看node节点

kubectl get nodes

部署集群网络

ansible-playbook 06.network.yml

kubectl get pod -n kube-system 

安装集群插件(dns, dashboard)

ansible-playbook 07.cluster-addon.yml

查看kube-system namespace下的服务

kubectl get svc -n kube-system

一步到位安装（上面七步可以直接用下面命令）

ansible-playbook 90.setup.yml

查看集群信息：

kubectl cluster-info

查看node/pod使用资源情况：

kubectl top node
kubectl top pod --all-namespaces

测试DNS

创建nginx service

kubectl run nginx --image=nginx --expose --port=80

创建busybox 测试pod

kubectl run busybox --rm -it --image=busybox /bin/sh //进入到busybox内部
nslookup nginx.default.svc.cluster.local //结果如下
Server: 10.68.0.2
Address: 10.68.0.2:53
Name: nginx.default.svc.cluster.local
Address: 10.68.9.156

 备份和恢复

[root@centos-04 ~]# cd 
[root@centos-04 ~]# kubectl run mysql --image=mysql:5.6 --expose --port=3306 （自动创建mysql的service和mysql的deployment）

创建备份目录

[root@centos-04 ~]# mkdir -p /backup/k8s
[root@centos-04 ~]# 

备份etcd数据

[root@centos-04 ~]# ETCDCTL_API=3 etcdctl snapshot save /backup/k8s/snapshot.db

备份ca证书

[root@centos-04 ~]# cp /etc/kubernetes/ssl/ca* /backup/k8s/
[root@centos-04 ~]# 

模拟集群崩溃

deploy节点执行 ansible-playbook /etc/ansible/99.clean.yml

恢复步骤如下（在deploy节点）：

恢复ca证书（我靠有问题，完了完了，我们用户一键安装重新安装一遍吧）

mkdir -p /etc/kubernetes/ssl
cp /backup/k8s/ca* /etc/kubernetes/ssl/

[root@centos-04 ~]# cp /backup/k8s/ca* /etc/kubernetes/ssl/
cp: 无法获取"/backup/k8s/ca*" 的文件状态(stat): 没有那个文件或目录
[root@centos-04 ~]# 

ansible-playbook 90.setup.yml

检查etcd是否成功

for ip in 130 131 132 ; do ETCDCTL_API=3 etcdctl --endpoints=https://192.168.242.$ip:2379 --
cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done

检查master节点的集群状态

kubectl get componentstatus 

查看node节点

kubectl get nodes

查看kube-system namespace下的服务

kubectl get svc -n kube-system

查看集群信息

kubectl cluster-info

创建nginx

[root@centos-04 ansible]# kubectl run nginx --image=nginx --expose --port=80

创建mysql

[root@centos-04 ansible]# history |grep run |grep mysql
  935  kubectl run mysql --image=mysql:5.6 --expose --port=3306
  958  history |grep run |grep mysql
[root@centos-04 ansible]#  kubectl run mysql --image=mysql:5.6 --expose --port=3306

查看pod所在的节点(可以看到ip)

kubectl get  pod   -o wide

删除某个节点

创建mysql失败查看logs发现需要创建初始化密码

 我们重新备份

执行clear

 ansible-playbook /etc/ansible/99.clean.yml 

恢复ca证书

mkdir -p /etc/kubernetes/ssl
cp /backup/k8s/ca* /etc/kubernetes/ssl/

重建集群

cd /etc/ansible
ansible-playbook 01.prepare.yml
ansible-playbook 02.etcd.yml
ansible-playbook 03.docker.yml
ansible-playbook 04.kube-master.yml
ansible-playbook 05.kube-node.yml

恢复etcd数据

停止服务

ansible etcd -m service -a 'name=etcd state=stopped'

清空文件

ansible etcd -m file -a 'name=/var/lib/etcd/member/ state=absent'

登录所有的etcd节点，参照本etcd节点/etc/systemd/system/etcd.service的服务文件，替换如下{{}}中变量后执行（在每台机器执行下面的命令都需要修改对应的红色部分，改为对应的etcd* 和对应的ip）

cd /backup/k8s/
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db 
 --name etcd1 
 --initialcluster
etcd1=https://192.168.242.130:2380,etcd2=https://192.168.242.131:2380,etcd3=https://192.168.242.132:2380

 --initial-cluster-token etcd-cluster-0 
 --initial-advertise-peer-urls https://192.168.111.128:2380

将128服务器的backup目录拷贝到129 130服务器

 执行上面的步骤后，会生成{{ NODE_NAME }}.etcd目录（三台机器都执行下面的对应命令）

cp -r etcd1.etcd/member /var/lib/etcd/
systemctl restart etcd

检查是否都好了

在deploy节点重建网络

ansible-playbook /etc/ansible/tools/change_k8s_network.yml

不想手动恢复，可以用ansible自动恢复 需要一键备份

ansible-playbook /etc/ansible/23.backup.yml

检查/etc/ansible/roles/cluster-backup/files目录下是否有文件

tree /etc/ansible/roles/cluster-backup/files/ //如下
├── ca # 集群CA 相关备份
│ ├── ca-config.json
│ ├── ca.csr
│ ├── ca-csr.json
│ ├── ca-key.pem
│ └── ca.pem
├── hosts # ansible hosts备份
│ ├── hosts # 最近的备份
│ └── hosts-201807231642
├── readme.md
└── snapshot # etcd 数据备份
 ├── snapshot-201807231642.db
 └── snapshot.db # 最近的备份

模拟故障：

ansible-playbook /etc/ansible/99.clean.yml

修改文件/etc/ansible/roles/cluster-restore/defaults/main.yml，指定要恢复的etcd快照备份，如果不修改就是最新的一次

恢复操作：

ansible-playbook /etc/ansible/24.restore.yml
ansible-playbook /etc/ansible/tools/change_k8s_network.yml