Phenomenon
overlay2 not such file or directory.
overlay2 not such file or directory.
network not such file or directory.
docker run ****
/usr/bin/docker-current: open /data0/docker-data/tmp/GetImageBlob742939759: no such file or directory.
Job for docker.service failed
[root@localhost file-center-wf-monitor]# systemctl stopstop docker
[root@localhost file-center-wf-monitor]# service docker start
Redirecting to /bin/systemctl start docker.service
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root@localhost file-center-wf-monitor]# journalctl -xe
Apr 08 16:46:56 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 08 16:46:56 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Apr 08 16:46:56 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Apr 08 16:46:56 localhost.localdomain systemd[1]: docker.service failed.
Apr 08 16:46:56 localhost.localdomain polkitd[378650]: Unregistered Authentication Agent for unix-process:615774:734710755 (system bus name :1.130780, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
[root@localhost file-center-wf-monitor]# rm -rf /data0/docker_images/*
rm: cannot remove ‘/data0/docker_images/containers’: Device or resource busy
[root@localhost file-center-wf-monitor]# mount | grep /data0/docker_images
/dev/sda7 on /data0/docker_images/containers type ext4 (rw,relatime,data=ordered)
/dev/sda7 on /data0/docker_images/overlay type ext4 (rw,relatime,data=ordered)
[root@localhost file-center-wf-monitor]# umount /data0/docker_images/containers
[root@localhost file-center-wf-monitor]# rm -rf /data0/docker_images
init/merged: invalid argument.
[root@localhost file-center-wf-monitor]# service docker start
Error response from daemon: Cannot kill container file-center-wf-monitor-20200408: No such container: file-center-wf-monitor-20200408
Error response from daemon: No such container: file-center-wf-monitor-20200408
/usr/bin/docker-current: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/dd9b026441e21f3b74bd00dbb9cba667129b8aba342413fa9fa56f68202b582b-init/merged: invalid argument.
See '/usr/bin/docker-current run --help'.
[root@localhost data0]# systemctl status docker.service
● docker.service - Docker Application Container Engine
{
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2020-04-08 16:54:09 CST; 6s ago
Docs: http://docs.docker.com
Process: 617751 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 617751 (code=exited, status=1/FAILURE)
Apr 08 16:54:08 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Apr 08 16:54:08 localhost.localdomain dockerd-current[617751]: time="2020-04-08T16:54:08.821786068+08:00" level=info msg="libcontainerd: new containerd process, pid: 617763"
Apr 08 16:54:09 localhost.localdomain dockerd-current[617751]: Error starting daemon: error initializing graphdriver: invalid argument
Apr 08 16:54:09 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 08 16:54:09 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Apr 08 16:54:09 localhost.localdomain systemd[1]: Unit docker.service entered failed state.
Apr 08 16:54:09 localhost.localdomain systemd[1]: docker.service failed.
solution
- 停止docker
[root@localhost file-center-wf-monitor]# systemctl stopstop docker
- 重装docker
-
卸载docker及关键依赖
yum list installed | grep docker yum remove docker-engine.x86_64 yum remove docker-engine-selinux.noarch yum remove docker-engine.x86_64 docker-client.x86_64 docker-common.x86_64 yum remove container-selinux.noarch- 删除docker 相关文件
rm -rf /var/lib/docker/ rm -rf /var/lib/docker rm -rf /data0/docker_images/overlay2/* rm -rf /data0/docker_images/* rm -rf /etc/sysconfig/docker* rm -rf /etc/systemd/system/docker.service.d- 不卸载会导致安装不上
Package(s) docker available, but not installed.- 不删除可能导致卸载不完全
-
安装docker
yum -y install docker
-
- 调整配置
[root@localhost file-center-wf-monitor]# vim /etc/docker/daemon.json
{
"graph": "/data0/docker_images",
"storage-driver": "overlay"
}
- 重载服务
systemctl daemon-reload
- 重启docker
service [start|restart] docker
- 查看docker状态
systemctl status docker
conclusion
- 本次问题根源是因为机器上某目录不可用，推测是由于历史docker版本残留导致
- 调整docker graph后，问题解决。
- 本文记录为解决当前问题过程中，所遇到的一系列问题和对应的操作。
- yum安装docker，如果机器比较干净，安装会很顺利；如果历史安装过docker，会出现一系列问题。可以分别卸载docker的子package，并手动清除残留信息解决。
- docker 镜像存储路径一定调整到系统数据盘，不然分分钟打满系统盘。
- journalctl -xe 可以看到service信息（docker/mesos）
- docker invalid argument不一定是启动的参数错误，一般不必调整
/etc/systemd/system/docker.service.d/storage.conf、/etc/sysconfig/docker-storage、/etc/sysconfig/docker和/etc/docker/daemon.json等文件。 /etc/sysconfig/docker配置可以关闭docker的selinux模式
# OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
OPTIONS='--log-driver=journald --signature-verification=false'
/etc/sysconfig/docker-storage配置可以调整docker存储（挂载目录）模式
DOCKER_STORAGE_OPTIONS=
/etc/sysconfig/docker-network配置可以调整docker网络模式- 通过
/etc/systemd/system/docker.service.d/***.conf下的对应文件进行配置，同样生效 - docker启动时，本地目录权限、状态、挂载都应确认清楚，避免发生异常。