openshift上使用devicemapper

环境：openshift v3.6.173.0.5

openshift上devicemapper与官方文档中的描述略有不同，在官方文档的描述中，容器使用的lvm文件系统挂载在/var/lib/devicemapper下，因此可以在/var/lib/devicemapper/metadata中找到容器对应的devicemapper的配置信息，在/var/lib/devicemapper/mnt中可以看到容器的文件系统，描述如下：

The /var/lib/docker/devicemapper/metadata/ directory contains metadata about the Devicemapper configuration itself and about each image and container layer that exist. The devicemapper storage driver uses snapshots, and this metadata include information about those snapshots. These files are in JSON format.

The /var/lib/devicemapper/mnt/ directory contains a mount point for each image and container layer that exists. Image layer mount points are empty, but a container’s mount point shows the container’s filesystem as it appears from within the container.

在openshfit中使用docker inspect查看一个容器的devicemapper信息如下，使用的块设备为 docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

"GraphDriver": {
    "Name": "devicemapper",
    "Data": {
        "DeviceId": "29",
        "DeviceName": "docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f",
        "DeviceSize": "10737418240"
    }
},

但在/var/lib/docker/devicemapper/mnt中该容器对应的文件挂载为空，系统mount命令也查找不到该容器对应的块设备挂载信息

# pwd
/var/lib/docker/devicemapper
# du -d 2|grep 13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f
0       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f-init
0       ./mnt/13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f

查看该容器对应的进程，pid为19422

# docker inspect 17ba06eb4578|grep Pid
  "Pid": 19422,
  "PidMode": "",
  "PidsLimit": 0,

在/proc/19422/ns中可以看到其命名空间相关的信息，对比pid 为1的mnt ns(mnt -> mnt:[4026531840])，可以看到该容器与根进程不属于同一个mount 命名空间，因此在根进程所在的mount命名空间中无法查看到该容器的挂载信息

# ll
total 0
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 ipc -> ipc:[4026532228]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 mnt -> mnt:[4026532521]
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 net -> net:[4026532231]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 pid -> pid:[4026532523]
lrwxrwxrwx. 1 1000010000 root 0 Apr 12 09:43 user -> user:[4026531837]
lrwxrwxrwx. 1 1000010000 root 0 Apr 11 19:47 uts -> uts:[4026532522]

使用nsenter -t 19422 -m mnt -u命令进入到进程所在的mount和uts命名空间，使用mount命令可以看到容器的块设备挂载到了该mount命名空间的根目录

/dev/mapper/docker-253:0-101504694-13339a03e1b2fc605c83e915a439d8f87131b9e01d599750298a5eada849ae5f on / type xfs (rw,relatime,context="system_u:object_r:container_file_t:s0:c2,c3",nouuid,attr2,inode64,sunit=1024,swidth=1024,noquota)

当然也可以在/proc/19422/mounts和/proc/19422/mountinfo中看到与该进程相关的mount信息，其中也包括容器的块设备挂载信息

使用lsns可以看系统中的命名空间与进程的对应关系，下面列出了pid为19422的相关命名空间，NPROCS表示该命名空间下面的进程数目，PID表示该命名空间下的最小PID值。可以看出19422与根进程属于同一个user命名空间。其中mnt命名空间中有2个进程，一个是19422，另一个是19422的父进程(此处为java)。因此在容器异常退出后，可以通过进入未退出进程的命名空间定位问题。

[root@lab-node1 proc]# lsns -p 19422
        NS TYPE  NPROCS   PID USER       COMMAND
4026531837 user     338     1 root       /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532228 ipc        3 18413 1001       /usr/bin/pod
4026532231 net        3 18413 1001       /usr/bin/pod
4026532521 mnt        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
4026532522 uts        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru
4026532523 pid        2 19422 1000010000 /bin/sh /opt/eap/bin/standalone.sh -Djavax.net.ssl.keyStore=/opt/hawkular/auth/hawkular-metrics.keystore -Djavax.net.ssl.tru

参考：

https://xuxinkun.github.io/2019/04/02/deviemapper-docker/

https://docs.docker.com/engine/reference/commandline/dockerd/