k8s集群新加入了一个节点
部分pod起不来，查看详情，定位到glusterfs挂载的问题上
kubectl describe pod aaaaa
ssh登录到目标机器，直接敲命令行挂载，同样挂载失败
查看挂载异常日志
# more /var/log/glusterfs/home-glusterfs-service.log
[2020-11-13 02:54:08.607196] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2020-11-13 02:54:08.613884] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-11-13 02:54:08.637663] W [MSGID: 101095] [xlator.c:213:xlator_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.15/xlator/features/utime.so: cannot open shared object file: No such file or director
y
[2020-11-13 02:54:08.637692] E [MSGID: 101002] [graph.y:213:volume_type] 0-parser: Volume 'vol_412ecd91390abff119945310ccef6214-utime', line 61: type 'features/utime' is not valid or not found on th
is machine
[2020-11-13 02:54:08.637710] E [MSGID: 101019] [graph.y:321:volume_end] 0-parser: "type" not specified for volume vol_412ecd91390abff119945310ccef6214-utime
[2020-11-13 02:54:08.637854] E [MSGID: 100026] [glusterfsd.c:2358:glusterfs_process_volfp] 0-: failed to construct the graph
[2020-11-13 02:54:08.638143] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x563e062c6092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x150) [0x563e06
2bfda0] -->/lib64/libglusterfs.so.0(glusterfs_graph_destroy+0x84) [0x7fd3f51ce754] ) 0-graph: invalid argument: graph [Invalid argument]
[2020-11-13 02:54:08.638204] W [glusterfsd.c:1375:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x563e062c6092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x163) [0x563e062b
fdb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x563e062bf32b] ) 0-: received signum (-1), shutting down
[2020-11-13 02:54:08.638255] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting '/home/cuidapeng/glusterfs/buzz-service'.
[2020-11-13 02:54:08.649588] I [fuse-bridge.c:5857:fini] 0-fuse: Closing fuse connection to '/home/cuidapeng/glusterfs/buzz-service'.
[2020-11-13 02:54:08.649761] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fd3f3ff6e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563e062bf4b5] -->/usr/sbi
n/glusterfs(cleanup_and_exit+0x6b) [0x563e062bf32b] ) 0-: received signum (15), shutting down
搜索到相关问题
https://github.com/gluster/glusterfs/issues/658
Please use gluster volume set $volname ctime off. This happens when all servers are upgraded, but clients are of old version (<5.0). We will see how to handle this without this check, but for now, I see the only option is to set this option.
版本不对，该节点上的gluster版本过低，升级重新安装最新版本即可
# cat /etc/yum.repos.d/CentOS-Gluster-6.repo
# CentOS-Gluster-6.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Storage for more
# information
[centos-gluster6]
name=CentOS-$releasever - Gluster 6
mirrorlist=http://mirrorlist.centos.org?arch=$basearch&release=$releasever&repo=storage-gluster-6
#baseurl=http://mirror.centos.org/$contentdir/$releasever/storage/$basearch/gluster-6/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
[centos-gluster6-test]
name=CentOS-$releasever - Gluster 6 Testing
baseurl=http://buildlogs.centos.org/centos/$releasever/storage/$basearch/gluster-6/
gpgcheck=0
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
yum install glusterfs-fuse