[Thu Jan 02 09:52:26.954370 2020] [core:notice] [pid 341] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Jan 02 09:52:26.955608 2020] [suexec:notice] [pid 341] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 02 09:52:26.985313 2020] [so:warn] [pid 341] AH01574: module proxy_module is already loaded, skipping
[Thu Jan 02 09:52:26.985368 2020] [so:warn] [pid 341] AH01574: module proxy_ajp_module is already loaded, skipping
[Thu Jan 02 09:52:26.985381 2020] [so:warn] [pid 341] AH01574: module proxy_balancer_module is already loaded, skipping
[Thu Jan 02 09:52:26.985426 2020] [so:warn] [pid 341] AH01574: module proxy_connect_module is already loaded, skipping
[Thu Jan 02 09:52:26.985444 2020] [so:warn] [pid 341] AH01574: module proxy_ftp_module is already loaded, skipping
[Thu Jan 02 09:52:26.985456 2020] [so:warn] [pid 341] AH01574: module proxy_http_module is already loaded, skipping
[Thu Jan 02 09:52:26.985466 2020] [so:warn] [pid 341] AH01574: module proxy_scgi_module is already loaded, skipping
[Thu Jan 02 09:52:26.985477 2020] [so:warn] [pid 341] AH01574: module slotmem_shm_module is already loaded, skipping
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.13.x.x Set the 'ServerName' directive globally to suppress this message
[Thu Jan 02 09:52:26.986823 2020] [auth_digest:notice] [pid 341] AH01757: generating secret for digest authentication ...
[Thu Jan 02 09:52:26.989025 2020] [lbmethod_heartbeat:notice] [pid 341] AH02282: No slotmem from mod_heartmonitor
[Thu Jan 02 09:52:27.016849 2020] [core:warn] [pid 341] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jan 02 09:52:27.016996 2020] [core:emerg] [pid 341] (28)No space left on device: AH00023: Couldn't create the mpm-accept mutex
【故障原因】
No space left on device，是因为进程开始申请的一块共享内存区不足，导致apache和zabbix信号量及它们之间的子进程通信异常。
linux系统的ipc信号量造成的，apache启动时，会创建很多子进程。信号量和子进程在共享内存区内进行通信。共享内存区是系统出于多个进程之间通讯的考虑，而预留的的一块内存区。
【排查过程】
[root@pmsmap01 zabbix]# ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 18350080 zabbix 600 14
0x00000000 18153473 apache 600 1
0x00000000 18186242 apache 600 1
0x00000000 18219011 apache 600 1
0x00000000 18251780 apache 600 1
0x00000000 655365 zabbix 600 14
0x00000000 18284550 apache 600 1
0x00000000 11042823 zabbix 600 14
0x00000000 11075592 zabbix 600 14
0x00000000 11108361 zabbix 600 14
0x00000000 11141130 zabbix 600 14
0x00000000 11173899 zabbix 600 14
0x00000000 11206668 zabbix 600 14
0x00000000 11239437 zabbix 600 14
0x00000000 11272206 zabbix 600 14
0x00000000 11304975 zabbix 600 14
0x00000000 11337744 zabbix 600 14
0x00000000 11370513 zabbix 600 14
0x00000000 11403282 zabbix 600 14
0x00000000 11436051 zabbix 600 14
0x00000000 11468820 zabbix 600 14
0x00000000 11501589 zabbix 600 14
0x00000000 11534358 zabbix 600 14
0x00000000 11567127 zabbix 600 14
0x00000000 11599896 zabbix 600 14
0x00000000 11632665 zabbix 600 14
0x00000000 11665434 zabbix 600 14
0x00000000 11698203 zabbix 600 14
0x00000000 11730972 zabbix 600 14
0x00000000 11763741 zabbix 600 14
0x00000000 11796510 zabbix 600 14
0x00000000 11829279 zabbix 600 14
0x00000000 11862048 zabbix 600 14
0x00000000 11894817 zabbix 600 14
0x00000000 11927586 zabbix 600 14
0x00000000 11960355 zabbix 600 14
0x00000000 11993124 zabbix 600 14
0x00000000 12025893 zabbix 600 14
0x00000000 12058662 zabbix 600 14
0x00000000 12091431 zabbix 600 14
0x00000000 12124200 zabbix 600 14
0x00000000 12156969 zabbix 600 14
0x00000000 12189738 zabbix 600 14
0x00000000 12222507 zabbix 600 14
0x00000000 12255276 zabbix 600 14
0x00000000 12288045 zabbix 600 14
0x00000000 12320814 zabbix 600 14
0x00000000 12353583 zabbix 600 14
0x00000000 12386352 zabbix 600 14
0x00000000 12419121 zabbix 600 14
0x00000000 12451890 zabbix 600 14
0x00000000 12484659 zabbix 600 14
0x00000000 12517428 zabbix 600 14
0x00000000 12550197 zabbix 600 14
0x00000000 12582966 zabbix 600 14
0x00000000 12615735 zabbix 600 14
0x00000000 12648504 zabbix 600 14
0x00000000 12681273 zabbix 600 14
0x00000000 12714042 zabbix 600 14
0x00000000 12746811 zabbix 600 14
0x00000000 12779580 zabbix 600 14
0x00000000 12812349 zabbix 600 14
0x00000000 12845118 zabbix 600 14
0x00000000 12877887 zabbix 600 14
0x00000000 12910656 zabbix 600 14
0x00000000 12943425 zabbix 600 14
0x00000000 12976194 zabbix 600 14
0x00000000 13008963 zabbix 600 14
0x00000000 13041732 zabbix 600 14
0x00000000 13074501 zabbix 600 14
0x00000000 13107270 zabbix 600 14
0x00000000 13140039 zabbix 600 14
0x00000000 13172808 zabbix 600 14
0x00000000 13205577 zabbix 600 14
0x00000000 13238346 zabbix 600 14
0x00000000 13271115 zabbix 600 14
0x00000000 13303884 zabbix 600 14
0x00000000 13336653 zabbix 600 14
0x00000000 13369422 zabbix 600 14
0x00000000 13402191 zabbix 600 14
0x00000000 13434960 zabbix 600 14
0x00000000 13467729 zabbix 600 14
0x00000000 13500498 zabbix 600 14
0x00000000 13533267 zabbix 600 14
0x00000000 13566036 zabbix 600 14
0x00000000 13598805 zabbix 600 14
0x00000000 13631574 zabbix 600 14
0x00000000 13664343 zabbix 600 14
0x00000000 13697112 zabbix 600 14
0x00000000 13729881 zabbix 600 14
0x00000000 13762650 zabbix 600 14
0x00000000 13795419 zabbix 600 14
0x00000000 13828188 zabbix 600 14
0x00000000 13860957 zabbix 600 14
0x00000000 13893726 zabbix 600 14
0x00000000 13926495 zabbix 600 14
0x00000000 13959264 zabbix 600 14
0x00000000 13992033 zabbix 600 14
0x00000000 14024802 zabbix 600 14
0x00000000 14057571 zabbix 600 14
0x00000000 14090340 zabbix 600 14
0x00000000 14123109 zabbix 600 14
0x00000000 14155878 zabbix 600 14
0x00000000 14188647 zabbix 600 14
0x00000000 14221416 zabbix 600 14
0x00000000 14254185 zabbix 600 14
0x00000000 14286954 zabbix 600 14
0x00000000 14319723 zabbix 600 14
0x00000000 14352492 zabbix 600 14
0x00000000 14385261 zabbix 600 14
0x00000000 14418030 zabbix 600 14
0x00000000 14450799 zabbix 600 14
0x00000000 14483568 zabbix 600 14
0x00000000 14516337 zabbix 600 14
0x00000000 14549106 zabbix 600 14
0x00000000 14581875 zabbix 600 14
0x00000000 14614644 zabbix 600 14
0x00000000 14647413 zabbix 600 14
0x00000000 14680182 zabbix 600 14
0x00000000 14712951 zabbix 600 14
0x00000000 14745720 zabbix 600 14
0x00000000 14778489 zabbix 600 14
0x00000000 14811258 zabbix 600 14
0x00000000 14844027 zabbix 600 14
0x00000000 14876796 zabbix 600 14
0x00000000 14909565 zabbix 600 14
0x00000000 14942334 zabbix 600 14
0x00000000 14975103 zabbix 600 14
【操作命令】
[root@zabbix]# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
[root@zabbix]# ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
【再启动服务】
[root@zabbix]# systemctl start zabbix-agent