• Kerbernetes的volume应用进阶


                Kerbernetes的volume应用进阶

                                         作者:尹正杰

    版权声明:原创作品,谢绝转载!否则将追究法律责任。 

    一.PV及PVC概述

    1>.什么是persistentVolume(持久卷,简称PV)

    [root@test110.yinzhengjie.org.cn ~]# cat /etc/exports
    /yinzhengjie/data/volume1    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume2    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume3    172.200.0.0/21(rw,no_root_squash)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# cat /etc/exports
      所谓的PV其实就是存储卷的意思,它可并不是和存储系统一一对应的关系哟,它对应的是存储系统的一个存储空间对应一个pv。

      我们以NFS的配置文件为例(参考上面的"/etc/exports"文件),"/yinzhengjie/data/volume1"其实对应的就是一个pv,"/yinzhengjie/data/volume2"和"/yinzhengjie/data/volume3"均各自对应一个pv。

    2>.什么是persistentVolumeClaim(简称PVC)

      我们知道Kubernetes集群有两个级别,一种是名称空间,一种是集群资源。Pod属于名称空间的资源,而pv则是集群资源,如果名称空间的Pod想要使用pv,则需要将pv注册(说白了就是请求占用pv)到和该Pod相同的名称空间中,我们称之为persistentVolumeClaim(简称PVC)。
    
      所谓的PVC我们可以理解为PV的一个客户端,pvc和pv之间没有直接联系,一个PVC可以绑定一个pv,此时pvc才会该pv建立关系,该pv就处于binding状态。与此同时,一个名称空间的pvc占用了某个pv,则该pv不能被其它名称的空间占用。
     
      如果一个pv没有被任何pvc所绑定,那么该pv就处于available。

      温馨提示:
        我们在Pod上使用persistentVolumeClaim类型的存储卷来使用pvc,我们在Pod上定义了pvc的存储卷容器是不能直接使用它的,Pod中的容器需要使用"
    volumeMounts"字段去挂载存储卷。
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pods.spec.volumes.persistentVolumeClaim
    KIND:     Pod
    VERSION:  v1
    
    RESOURCE: persistentVolumeClaim <Object>
    
    DESCRIPTION:
         PersistentVolumeClaimVolumeSource represents a reference to a
         PersistentVolumeClaim in the same namespace. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
    
         PersistentVolumeClaimVolumeSource references the user's PVC in the same
         namespace. This volume finds the bound PV and mounts that volume for the
         pod. A PersistentVolumeClaimVolumeSource is, essentially, a wrapper around
         another type of volume that is owned by someone else (the system).
    
    FIELDS:
       claimName    <string> -required-
         ClaimName is the name of a PersistentVolumeClaim in the same namespace as
         the pod using this volume. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
    
       readOnly    <boolean>
         Will force the ReadOnly setting in VolumeMounts. Default false.
    
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pods.spec.volumes.persistentVolumeClaim

    3>.pvc的访问模式(accessModes)取决于pv

      一般存储系统的访问模型大致分为以下三种:
        单路读写(ReadWriteOnce,简称为RWO)
        多路只读(ReadOnlyMany,简称为ROM)
        多路读写(ReadWriteMany,简称为RWM)
    
      同一个pvc是否可以同时被多个Pod同时使用呢?
        这取决于pv是否支持多路读写(比如NFS就支持多路客户端同时读写访问,而Iscsi这种块级别的存储则不支持多路读写)。如果pv支持多路读写,那么与之绑定的pvc也支持多路读写,也就是说支持多个Pod同时绑定到同一个pvc上,反之则反。
    
      博主推荐阅读:
        https://kubernetes.io/docs/concepts/storage/persistent-volumes/#lifecycle-of-a-volume-and-claim

    4>.pv的回收(reClaim)策略

      pv除了上面我们所说的binding和available两种状态外还有两种就是回收(reClaim)状态。
    
      pv的回收(reClaim)策略有以下三种:
        Recycle(deprecated,已弃用):
          删除pvc时,只把pv上的数据删除,也就是说并没有把pv资源删除,该策略已被Google废弃,在后续版本可能会被移除,因此不推荐使用。
        Delete:
          删除PVC时,也会同时删除pv资源。
        Retain:
          删除pvc时,pv和pv上的数据军均不删除,对于重要的数据生产环境推荐使用这种策略。
     
      博主推荐阅读:
        https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes

    5>.PVC的生命周期

      pv的供给(Provisioning)方式:
        static:
          说白了就得咱们人工手动创建pv,比如NFS等
        dynamic:
          说白了就是无序人工介入,存储系统自动就支持pv的创建啦,比如Cephfs等,它是需要借助于存储类(StorageClass,简称SC),当创建pvc时,pvc会向SC请求,SC已经内置了如何连接pv的方式。

      pvc的生命周期分为四种状态:     Provisioning(即PV是如何创建的,NFS默认只支持静态创建pv,而cephfs是支持动态创建pv)     Binding(绑定了PV)     Using(Pod在使用pvc)     Reclaiming(回收状态)。   博主推荐阅读:     https:
    //kubernetes.io/docs/concepts/storage/dynamic-provisioning/

    二.持久卷persistentVolumeClaim(简称PVC)存储实战案例

    1>.构建NFS网络文件系统

    [root@test110.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirrors.tuna.tsinghua.edu.cn
     * updates: mirrors.tuna.tsinghua.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (4/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (5/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (6/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (7/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (8/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (9/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (10/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
    (11/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (12/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (14/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.9 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@test110.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
    mkdir: created directory ‘/yinzhengjie/data’
    mkdir: created directory ‘/yinzhengjie/data/volume1’
    mkdir: created directory ‘/yinzhengjie/data/volume2’
    mkdir: created directory ‘/yinzhengjie/data/volume3’
    mkdir: created directory ‘/yinzhengjie/data/volume4’
    mkdir: created directory ‘/yinzhengjie/data/volume5’
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/ -R
    /yinzhengjie/data/:
    total 0
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume1
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume2
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume4
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5
    
    /yinzhengjie/data/volume1:
    total 0
    
    /yinzhengjie/data/volume2:
    total 0
    
    /yinzhengjie/data/volume3:
    total 0
    
    /yinzhengjie/data/volume4:
    total 0
    
    /yinzhengjie/data/volume5:
    total 0
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# cat /etc/exports
    /yinzhengjie/data/volume1    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume2    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume3    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume4    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume5    172.200.0.0/21(rw,no_root_squash)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/
    total 0
    drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume2
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume3
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume4
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume5
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# exportfs -rav
    exporting 172.200.0.0/21:/yinzhengjie/data/volume5
    exporting 172.200.0.0/21:/yinzhengjie/data/volume4
    exporting 172.200.0.0/21:/yinzhengjie/data/volume3
    exporting 172.200.0.0/21:/yinzhengjie/data/volume2
    exporting 172.200.0.0/21:/yinzhengjie/data/volume1
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# systemctl status nfs
    ● nfs-server.service - NFS server and services
       Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
       Active: inactive (dead)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl start nfs
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl enable nfs
    Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl status nfs
    ● nfs-server.service - NFS server and services
       Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
      Drop-In: /run/systemd/generator/nfs-server.service.d
               └─order-with-mounts.conf
       Active: active (exited) since Mon 2020-02-10 06:08:52 CST; 6s ago
     Main PID: 6116 (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/nfs-server.service
    
    Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Starting NFS server and services...
    Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Started NFS server and services.
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ss -ntl | grep 2049
    LISTEN     0      64           *:2049                     *:*                  
    LISTEN     0      64          :::2049                    :::*                  
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# iptables -vnL
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# iptables -vnL -t nat
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl start nfs
    nfs无法启动:
      如果你nfs启动失败很可能是由于rpcbind服务没有启动导致的。因为nfs服务是依赖于rpc服务的,我的报错信息如下所示:
        Feb 10 18:36:52 test110.yinzhengjie.org.cn systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
    
    解决方案:
        经查阅资料是需要禁用ipv6,修改rpc的配置文件后并重启rpcbind服务,之后nfs可以成功启动啦。具体操作如下所示。
    
    [root@test110.yinzhengjie.org.cn ~]#  find /etc/ -name '*rpcbind.socket*'
    /etc/systemd/system/sockets.target.wants/rpcbind.socket
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/systemd/system/sockets.target.wants/rpcbind.socket
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# cat /etc/systemd/system/sockets.target.wants/rpcbind.socket
    [Unit]
    Description=RPCbind Server Activation Socket
    
    [Socket]
    ListenStream=/var/run/rpcbind.sock
    
    # RPC netconfig can't handle ipv6/ipv4 dual sockets
    BindIPv6Only=ipv6-only
    ListenStream=0.0.0.0:111
    ListenDatagram=0.0.0.0:111
    #ListenStream=[::]:111   #果然监听了ipv6地址,将这一行注释即可
    ListenDatagram=[::]:111
    
    [Install]
    WantedBy=sockets.target
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl daemon-reload 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl restart rpcbind.socket 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl start nfs
    [root@test110.yinzhengjie.org.cn ~]# 
    "rpcbind.socket failed to listen on sockets: Address family not supported by protocol"报错解决方案,如果NFS正常启动的小伙伴可忽略本条记录。

    2>.在K8S集群的每一个节点安装nfs客户端驱动

    [root@master200.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirror.bit.edu.cn
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (4/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (5/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (6/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (7/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (11/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                            |  50 kB  00:00:00     
    (12/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    (13/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         1.4 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node201.yinzhengjie.org.cn ~]#  yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (2/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (3/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (4/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (5/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (6/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (7/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (8/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (10/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                         |  28 kB  00:00:00     
    (11/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (13/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.2 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node201.yinzhengjie.org.cn ~]# 
    [root@node201.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node202.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirror.bit.edu.cn
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (3/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (4/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (5/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (6/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (7/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (8/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (9/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (10/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (11/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                            |  89 kB  00:00:00     
    (12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (14/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         1.9 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node202.yinzhengjie.org.cn ~]# 
    [root@node202.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node203.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (3/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (4/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (5/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (6/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (7/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (9/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                               |  90 kB  00:00:00     
    (10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (11/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
    (12/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                      | 8.9 kB  00:00:00     
    (13/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.6 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node203.yinzhengjie.org.cn ~]# 
    [root@node203.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@master200.yinzhengjie.org.cn ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount | grep mnt
    172.200.1.110:/yinzhengjie/data/volume1 on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.200.1.200,local_lock=none,addr=172.200.1.110)
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# df -h | grep mnt
    172.200.1.110:/yinzhengjie/data/volume1  1.6T  416M  1.6T   1% /mnt
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载,即验证可用性。

    3>.静态供给(Provisioning)创建pv(由于pv是集群资源,并不属于名称空间,因此在定义pv时我们不能指定名称空间哟~)

    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pv
    KIND:     PersistentVolume
    VERSION:  v1
    
    DESCRIPTION:
         PersistentVolume (PV) is a storage resource provisioned by an
         administrator. It is analogous to a node. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes
    
    FIELDS:
       apiVersion    <string>
         APIVersion defines the versioned schema of this representation of an
         object. Servers should convert recognized schemas to the latest internal
         value, and may reject unrecognized values. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
    
       kind    <string>
         Kind is a string value representing the REST resource this object
         represents. Servers may infer this from the endpoint the client submits
         requests to. Cannot be updated. In CamelCase. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
    
       metadata    <Object>
         Standard object's metadata. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
    
       spec    <Object>
         Spec defines a specification of a persistent volume owned by the cluster.
         Provisioned by an administrator. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
    
       status    <Object>
         Status represents the current information/status for the persistent volume.
         Populated by the system. Read-only. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
    
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pv
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pv.spec
    KIND:     PersistentVolume
    VERSION:  v1
    
    RESOURCE: spec <Object>
    
    DESCRIPTION:
         Spec defines a specification of a persistent volume owned by the cluster.
         Provisioned by an administrator. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
    
         PersistentVolumeSpec is the specification of a persistent volume.
    
    FIELDS:
       accessModes    <[]string>
         AccessModes contains all ways the volume can be mounted. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes
    
       awsElasticBlockStore    <Object>
         AWSElasticBlockStore represents an AWS Disk resource that is attached to a
         kubelet's host machine and then exposed to the pod. More info:
         https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
    
       azureDisk    <Object>
         AzureDisk represents an Azure Data Disk mount on the host and bind mount to
         the pod.
    
       azureFile    <Object>
         AzureFile represents an Azure File Service mount on the host and bind mount
         to the pod.
    
       capacity    <map[string]string>
         A description of the persistent volume's resources and capacity. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
    
       cephfs    <Object>
         CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
    
       cinder    <Object>
         Cinder represents a cinder volume attached and mounted on kubelets host
         machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
    
       claimRef    <Object>
         ClaimRef is part of a bi-directional binding between PersistentVolume and
         PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName
         is the authoritative bind between PV and PVC. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding
    
       csi    <Object>
         CSI represents storage that is handled by an external CSI driver (Beta
         feature).
    
       fc    <Object>
         FC represents a Fibre Channel resource that is attached to a kubelet's host
         machine and then exposed to the pod.
    
       flexVolume    <Object>
         FlexVolume represents a generic volume resource that is
         provisioned/attached using an exec based plugin.
    
       flocker    <Object>
         Flocker represents a Flocker volume attached to a kubelet's host machine
         and exposed to the pod for its usage. This depends on the Flocker control
         service being running
    
       gcePersistentDisk    <Object>
         GCEPersistentDisk represents a GCE Disk resource that is attached to a
         kubelet's host machine and then exposed to the pod. Provisioned by an
         admin. More info:
         https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
    
       glusterfs    <Object>
         Glusterfs represents a Glusterfs volume that is attached to a host and
         exposed to the pod. Provisioned by an admin. More info:
         https://examples.k8s.io/volumes/glusterfs/README.md
    
       hostPath    <Object>
         HostPath represents a directory on the host. Provisioned by a developer or
         tester. This is useful for single-node development and testing only!
         On-host storage is not supported in any way and WILL NOT WORK in a
         multi-node cluster. More info:
         https://kubernetes.io/docs/concepts/storage/volumes#hostpath
    
       iscsi    <Object>
         ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
         host machine and then exposed to the pod. Provisioned by an admin.
    
       local    <Object>
         Local represents directly-attached storage with node affinity
    
       mountOptions    <[]string>
         A list of mount options, e.g. ["ro", "soft"]. Not validated - mount will
         simply fail if one is invalid. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
    
       nfs    <Object>
         NFS represents an NFS mount on the host. Provisioned by an admin. More
         info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
    
       nodeAffinity    <Object>
         NodeAffinity defines constraints that limit what nodes this volume can be
         accessed from. This field influences the scheduling of pods that use this
         volume.
    
       persistentVolumeReclaimPolicy    <string>
         What happens to a persistent volume when released from its claim. Valid
         options are Retain (default for manually created PersistentVolumes), Delete
         (default for dynamically provisioned PersistentVolumes), and Recycle
         (deprecated). Recycle must be supported by the volume plugin underlying
         this PersistentVolume. More info:
         https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming
    
       photonPersistentDisk    <Object>
         PhotonPersistentDisk represents a PhotonController persistent disk attached
         and mounted on kubelets host machine
    
       portworxVolume    <Object>
         PortworxVolume represents a portworx volume attached and mounted on
         kubelets host machine
    
       quobyte    <Object>
         Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
    
       rbd    <Object>
         RBD represents a Rados Block Device mount on the host that shares a pod's
         lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
    
       scaleIO    <Object>
         ScaleIO represents a ScaleIO persistent volume attached and mounted on
         Kubernetes nodes.
    
       storageClassName    <string>
         Name of StorageClass to which this persistent volume belongs. Empty value
         means that this volume does not belong to any StorageClass.
    
       storageos    <Object>
         StorageOS represents a StorageOS volume that is attached to the kubelet's
         host machine and mounted into the pod More info:
         https://examples.k8s.io/volumes/storageos/README.md
    
       volumeMode    <string>
         volumeMode defines if a volume is intended to be used with a formatted
         filesystem or to remain in raw block state. Value of Filesystem is implied
         when not included in spec. This is a beta feature.
    
       vsphereVolume    <Object>
         VsphereVolume represents a vSphere volume attached and mounted on kubelets
         host machine
    
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain pv.spec
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/volumes/pv-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/volumes/pv-demo.yaml
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v2
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume2
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/volumes/pv-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    No resources found in default namespace.
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/volumes/pv-demo.yaml
    persistentvolume/pv-nfs-v2 created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Available                                   87s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE     VOLUMEMODE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Available                                   2m44s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/volumes/pv-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE    VOLUMEMODE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Available                                   3m4s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pv 
    Name:            pv-nfs-v2
    Labels:          storsys=nfs
    Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"storsys":"nfs"},"name":"pv-nfs-v2"},"spec":{"accessMo...
    Finalizers:      [kubernetes.io/pv-protection]
    StorageClass:    
    Status:          Available
    Claim:           
    Reclaim Policy:  Retain
    Access Modes:    RWO,ROX,RWX
    VolumeMode:      Filesystem
    Capacity:        5Gi
    Node Affinity:   <none>
    Message:         
    Source:
        Type:      NFS (an NFS mount that lasts the lifetime of a pod)
        Server:    172.200.1.110
        Path:      /yinzhengjie/data/volume2
        ReadOnly:  false
    Events:        <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pv

    4>.定义pvc(pvc属于名称空间资源,因此我们在定义pvc时可以指定名称空间哟~)

    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE    VOLUMEMODE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Available                                   3m4s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide                      #pvc没有绑定pv之前
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/volumes/pvc-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/volumes/pvc-demo.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: redis-data
      namespace: yinzhengjie-volume
    spec:
      accessModes:
        - ReadWriteOnce
      volumeMode: Filesystem
      resources:
        requests:
          storage: 3Gi
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/volumes/pvc-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/volumes/pvc-demo.yaml 
    persistentvolumeclaim/redis-data created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pvc -n yinzhengjie-volume
    NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    redis-data   Bound    pv-nfs-v2   5Gi        RWO,ROX,RWX                   66s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/volumes/pvc-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE    VOLUMEMODE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Available                                   3m4s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pvc -n yinzhengjie-volume -o wide
    NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE    VOLUMEMODE
    redis-data   Bound    pv-nfs-v2   5Gi        RWO,ROX,RWX                   101s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pvc -n yinzhengjie-volume
    Name:          redis-data
    Namespace:     yinzhengjie-volume
    StorageClass:  
    Status:        Bound
    Volume:        pv-nfs-v2
    Labels:        <none>
    Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"redis-data","namespace":"yinzhengjie-volume"},"spec...
                   pv.kubernetes.io/bind-completed: yes
                   pv.kubernetes.io/bound-by-controller: yes
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      5Gi
    Access Modes:  RWO,ROX,RWX
    VolumeMode:    Filesystem
    Mounted By:    <none>
    Events:        <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -n yinzhengjie-volume
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-volume/redis-data                           14m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pvc -n yinzhengjie-volume
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv  -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE   VOLUMEMODE
    pv-nfs-v2   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-volume/redis-data                           18m   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide                      #pvc绑定pv之后

    5>.创建Pod并应用pvc 

    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: redis-demo
      namespace: yinzhengjie-volume
      labels:
        app: redis-demo
    spec:
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - mountPath: /data
          name: redisdata
      volumes:
        - name: redisdata
          persistentVolumeClaim: 
            claimName: redis-data
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp   1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    pod/redis-demo created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          7s    10.244.1.25   node201.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          96s   10.244.1.25   node201.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pods redis-demo -n yinzhengjie-volume 
    Name:         redis-demo
    Namespace:    yinzhengjie-volume
    Priority:     0
    Node:         node201.yinzhengjie.org.cn/172.200.1.201
    Start Time:   Mon, 10 Feb 2020 03:26:02 +0800
    Labels:       app=redis-demo
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"redis-demo"},"name":"redis-demo","namespace":"yinzhengjie-vo...
    Status:       Running
    IP:           10.244.1.25
    IPs:
      IP:  10.244.1.25
    Containers:
      redis:
        Container ID:   docker://395ba89580fa126ee755358f2afe8b9922b2084304b3b61a88ded14526e3979c
        Image:          redis:alpine
        Image ID:       docker-pullable://redis@sha256:cb9783b1c39bb34f8d6572406139ab325c4fac0b28aaa25d5350495637bb2f76
        Port:           6379/TCP
        Host Port:      0/TCP
        State:          Running
          Started:      Mon, 10 Feb 2020 03:26:03 +0800
        Ready:          True
        Restart Count:  0
        Environment:    <none>
        Mounts:
          /data from redisdata (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from default-token-bxflv (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             True 
      ContainersReady   True 
      PodScheduled      True 
    Volumes:
      redisdata:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  redis-data
        ReadOnly:   false
      default-token-bxflv:
        Type:        Secret (a volume populated by a Secret)
        SecretName:  default-token-bxflv
        Optional:    false
    QoS Class:       BestEffort
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
    Events:
      Type    Reason     Age        From                                 Message
      ----    ------     ----       ----                                 -------
      Normal  Scheduled  <unknown>  default-scheduler                    Successfully assigned yinzhengjie-volume/redis-demo to node201.yinzhengjie.org.cn
      Normal  Pulled     108s       kubelet, node201.yinzhengjie.org.cn  Container image "redis:alpine" already present on machine
      Normal  Created    108s       kubelet, node201.yinzhengjie.org.cn  Created container redis
      Normal  Started    108s       kubelet, node201.yinzhengjie.org.cn  Started container redis
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe pods redis-demo -n yinzhengjie-volume

    6>.连接pod并创建测试数据

    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h     10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          3m33s   10.244.1.25   node201.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it redis-demo -n yinzhengjie-volume -- /bin/sh
    /data # 
    /data # redis-cli 
    127.0.0.1:6379> 
    127.0.0.1:6379> KEYS *
    (empty list or set)
    127.0.0.1:6379> 
    127.0.0.1:6379> SET myblog "https://www.cnblogs.com/yinzhengjie/"
    OK
    127.0.0.1:6379> 
    127.0.0.1:6379> KEYS *
    1) "myblog"
    127.0.0.1:6379> 
    127.0.0.1:6379> BGSAVE
    Background saving started
    127.0.0.1:6379> 
    127.0.0.1:6379> exit
    /data # 
    /data # ls -l
    total 4
    -rw-r--r--    1 redis    redis          142 Feb 10  2020 dump.rdb
    /data # 
    /data # exit
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume2 /mnt
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount | grep mnt
    172.200.1.110:/yinzhengjie/data/volume2 on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.200.1.200,local_lock=none,addr=172.200.1.110)
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# df -h | grep mnt
    172.200.1.110:/yinzhengjie/data/volume2  1.6T  416M  1.6T   1% /mnt
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# ll /mnt/
    total 4
    -rw-r--r-- 1 polkitd 1000 142 Feb 10  2020 dump.rdb
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it redis-demo -n yinzhengjie-volume -- /bin/sh

    7>.删除Pod并重新创建Pod验证数据是否丢失

    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          11m   10.244.1.25   node201.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl delete -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    pod "redis-demo" deleted
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp   1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl delete -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: redis-demo
      namespace: yinzhengjie-volume
      labels:
        app: redis-demo
    spec:
      nodeName: node203.yinzhengjie.org.cn
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - mountPath: /data
          name: redisdata
      volumes:
        - name: redisdata
          persistentVolumeClaim: 
            claimName: redis-data
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME    READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp   1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    pod/redis-demo created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h   10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          24s   10.244.3.19   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo-redis.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-volume -o wide
    NAME         READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
    myapp        1/1     Running   1          20h     10.244.3.14   node203.yinzhengjie.org.cn   <none>           <none>
    redis-demo   1/1     Running   0          2m49s   10.244.3.19   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it redis-demo -n yinzhengjie-volume -- /bin/sh
    /data # 
    /data # redis-cli 
    127.0.0.1:6379> 
    127.0.0.1:6379> KEYS *
    1) "myblog"
    127.0.0.1:6379> 
    127.0.0.1:6379> get myblog
    "https://www.cnblogs.com/yinzhengjie/"
    127.0.0.1:6379> 
    127.0.0.1:6379> 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it redis-demo -n yinzhengjie-volume -- /bin/sh

    8>.删除pvc注意事项

      如下图所示,当有Pods在使用pvc时,我们无法是无法手动删除pvc的,这是官方的提出的一种存储保护机制。
    
      博主推荐阅读:
        https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection

      想要删除pvc得确保没有Pods使用pvc资源,如果把正在使用pvc的所有Pods资源删除掉,此时就可以理解删除pvc了,如下图所示。
    
      删除pvc后,如果pvc的删除策略是"Retain",虽然删除pvc是成功了,但并不会删除pv资源及其所对应的存储系统中的数据,我们需要手动删除。

    三.使用存储类案例

    [root@master200.yinzhengjie.org.cn ~]# kubectl explain sc
    KIND:     StorageClass
    VERSION:  storage.k8s.io/v1
    
    DESCRIPTION:
         StorageClass describes the parameters for a class of storage for which
         PersistentVolumes can be dynamically provisioned. StorageClasses are
         non-namespaced; the name of the storage class according to etcd is in
         ObjectMeta.Name.
    
    FIELDS:
       allowVolumeExpansion    <boolean>
         AllowVolumeExpansion shows whether the storage class allow volume expand
    
       allowedTopologies    <[]Object>
         Restrict the node topologies where volumes can be dynamically provisioned.
         Each volume plugin defines its own supported topology specifications. An
         empty TopologySelectorTerm list means there is no topology restriction.
         This field is only honored by servers that enable the VolumeScheduling
         feature.
    
       apiVersion    <string>
         APIVersion defines the versioned schema of this representation of an
         object. Servers should convert recognized schemas to the latest internal
         value, and may reject unrecognized values. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
    
       kind    <string>
         Kind is a string value representing the REST resource this object
         represents. Servers may infer this from the endpoint the client submits
         requests to. Cannot be updated. In CamelCase. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
    
       metadata    <Object>
         Standard object's metadata. More info:
         https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
    
       mountOptions    <[]string>
         Dynamically provisioned PersistentVolumes of this storage class are created
         with these mountOptions, e.g. ["ro", "soft"]. Not validated - mount of the
         PVs will simply fail if one is invalid.
    
       parameters    <map[string]string>
         Parameters holds the parameters for the provisioner that should create
         volumes of this storage class.
    
       provisioner    <string> -required-
         Provisioner indicates the type of the provisioner.
    
       reclaimPolicy    <string>
         Dynamically provisioned PersistentVolumes of this storage class are created
         with this reclaimPolicy. Defaults to Delete.
    
       volumeBindingMode    <string>
         VolumeBindingMode indicates how PersistentVolumeClaims should be
         provisioned and bound. When unset, VolumeBindingImmediate is used. This
         field is only honored by servers that enable the VolumeScheduling feature.
    
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl explain sc
      存储类是为动态供给pv的,默认情况下,nfs是不支持动态供给的,但是可以通过插件实现让nfs也支持动态供给pv哟,参考连接:
        https://github.com/kubernetes-incubator/external-storage/tree/master/nfs/deploy/kubernetes
      
      博主推荐阅读:               https:
    //kubernetes.io/docs/concepts/storage/storage-classes/

  • 相关阅读:
    Execl获取真实行数
    日期工具类
    Eclipse & IDEA快捷键对比大全
    【转】IDEA导入Eclipse项目 详细步骤(亲自验证导入成功)
    京东的四大集团
    马云演讲视频
    阿里系的一些重要网站
    阿里系的几个电商网站
    腾讯几款QQ软件
    电脑版微信使用说明
  • 原文地址:https://www.cnblogs.com/yinzhengjie/p/12286111.html
Copyright © 2020-2023  润新知