kubernetes集群常见错误及解决方法

 记录一次kubernetes集群worker节点/var/log/messages系统日志错误信息如下：

Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9036_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9036_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9036_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9036_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9042_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9042_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9042_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9042_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9107_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9107_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9107_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9107_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9114_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9114_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9114_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9114_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9128_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9128_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9128_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9128_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9142_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9142_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Created slice libcontainer_9142_systemd_test_default.slice.
Jan  7 09:54:20 worker02 systemd: Removed slice libcontainer_9142_systemd_test_default.slice.
Jan  7 09:54:21 worker02 systemd: Created slice libcontainer_9148_systemd_test_default.slice.

 问题原因：

问题大致原因是由于将 cgroup-driver 设置为 systemd 后引起的，但是这些错误信息不会影响容器指标

https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L123

https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.2.0/troubleshoot/cgroup_driver.html
https://www.ibm.com/support/pages/recurring-messages-complain-scope-libcontainer-nnnnn-has-no-pids-refusing

解决方法：

## 方法1：
cat <<EOF >/etc/rsyslog.d/ignore-systemd-session-slice.conf
if ($programname == "systemd") and ($msg contains "_systemd_test_default.slice" or$msg contains "systemd-test-default-dependencies.scope") then { stop }
EOF
systemctl restart rsyslog.service

## 方法2：
cat <<EOF >/etc/rsyslog.d/ignore-systemd-session-slice.conf
:rawmsg, contains, "libcontainer" ~
EOF
systemctl restart rsyslog.service

　

 建议采用如下方式：　

从根源上解决：修改kubelet启动参数

KUBELET_EXTRA_ARGS=--kubelet-cgroups=/system.slice/kubelet.service --runtime-cgroups=/system.slice/docker.service

 修改kubelet服务启动的配置文件

cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=

#KUBELET_EXTRA_ARGS=--kubelet-cgroups=/system.slice/kubelet.service --runtime-cgroups=/system.slice/docker.service --feature-gates=LocalStorageCapacityIsolation=true 
#                   --kube-reserved-cgroup=/kubepods.slice --kube-reserved=cpu=500m,memory=500Mi,ephemeral-storage=1Gi 
#                   --system-reserved-cgroup=/system.slice --system-reserved=cpu=500m,memory=500Mi,ephemeral-storage=1Gi 
#                   --eviction-hard=memory.available<500Mi,nodefs.available<10%
#                   --max-pods=200

重启kubelet服务

systemctl restart kubelet.service

　　

2020.04.28记录

kubernetes集群worker节点/var/log/messages系统日志错误信息如下：

Apr 28 11:25:45 worker02 kernel: nfs4_reclaim_open_state: 6 callbacks suppressed
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Apr 28 11:25:45 worker02 kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!

　　

原因:文件句柄不释放，发生死锁

解决方法：优化worker节点的内核参数并重启nfs-server服务

有条件建议使用  GlusterFS 或者Ceph 等其他文件系统作为底层存储，性能、稳定，高可用都胜过 nfs

以下配置仅供参考，具体请根据服务器配置来自行优化：

1、调整sysctl.conf配置文件

cat /etc/sysctl.conf

net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
fs.aio-max-nr = 1048576
fs.file-max = 76724600
net.core.netdev_max_backlog = 10000
net.core.rmem_default = 262144 
# The default setting of the socket receive buffer in bytes.
net.core.rmem_max = 4194304 
# The maximum receive socket buffer size in bytes
net.core.wmem_default = 262144 
# The default setting (in bytes) of the socket send buffer.
net.core.wmem_max = 4194304 
# The maximum send socket buffer size in bytes.
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_keepalive_intvl = 20
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syncookies = 1 
# 开启SYN Cookies。当出现SYN等待队列溢出时，启用cookie来处理，可防范少量的SYN攻击
net.ipv4.tcp_timestamps = 1 
# 减少time_wait
net.ipv4.tcp_tw_recycle = 0 
# 如果=1则开启TCP连接中TIME-WAIT套接字的快速回收，但是NAT环境可能导致连接失败，建议服务端关闭它
net.ipv4.tcp_tw_reuse = 1 
# 开启重用。允许将TIME-WAIT套接字重新用于新的TCP连接
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 8192 65536 16777216
net.nf_conntrack_max = 1200000
net.netfilter.nf_conntrack_max = 1200000
vm.dirty_background_bytes = 409600000 
# 系统脏页到达这个值，系统后台刷脏页调度进程 pdflush（或其他） 自动将(dirty_expire_centisecs/100）秒前的脏页刷到磁盘
vm.dirty_expire_centisecs = 3000 
# 比这个值老的脏页，将被刷到磁盘。3000表示30秒。
vm.dirty_ratio = 95 
# 如果系统进程刷脏页太慢，使得系统脏页超过内存 95 % 时，则用户进程如果有写磁盘的操作（如fsync, fdatasync等调用），则需要主动把系统脏页刷出。
# 有效防止用户进程刷脏页，在单机多实例，并且使用CGROUP限制单实例IOPS的情况下非常有效。 
vm.dirty_writeback_centisecs = 100 
# pdflush（或其他）后台刷脏页进程的唤醒间隔， 100表示1秒。
vm.mmap_min_addr = 65536
vm.overcommit_memory = 0 
# 在分配内存时，允许少量over malloc, 如果设置为 1, 则认为总是有足够的内存，内存较少的测试环境可以使用 1 . 
vm.overcommit_ratio = 90 
# 当overcommit_memory = 2 时，用于参与计算允许指派的内存大小。
vm.swappiness = 0 
# 关闭交换分区
vm.zone_reclaim_mode = 0 
# 禁用 numa, 或者在vmlinux中禁止. 
net.ipv4.ip_local_port_range = 40000 65535 
# 本地自动分配的TCP, UDP端口号范围
fs.nr_open=20480000
# 单个进程允许打开的文件句柄上限
net.ipv4.tcp_max_syn_backlog = 16384
net.core.somaxconn = 16384

2、执行sysctl -p命令使参数立即生效

sysctl -p

3、修改文件最大打开数vi /etc/security/limits.conf

* soft nofile 1024000
* hard nofile 1024000
* soft nproc unlimited
* hard nproc unlimited
* soft core unlimited
* hard core unlimited
* soft memlock unlimited
* hard memlock unlimited

4、重启nfs-server服务端服务

systemctl restart nfs-server.service