• nginx日志统计相关命令和python脚本


    src=(s1,s2,s3,s3)

    for i in ${src[@]

     

     

    nginx访问量统计

    1.根据访问IP统计UV
    awk '{print $1}'  access.log|sort | uniq -c |wc -l

    嘉图的统计 awk '{print $1}'  /var/log/nginx/access.log|sort | uniq -c |wc -l
    2.统计访问URL统计PV
    awk '{print $7}' access.log|wc -l
    3.查询访问最频繁的URL
    awk '{print $7}' access.log|sort | uniq -c |sort -n -k 1 -r|more
    4.查询访问最频繁的IP
    awk '{print $1}' access.log|sort | uniq -c |sort -n -k 1 -r|more
    5.根据时间段统计查看日志
     cat  access.log| sed -n '/14/Mar/2015:21/,/14/Mar/2015:22/p'|more

    6.根据时间点统计查看日志.

    [root@hemlock nginx]#  sed -n '/22/Apr/2016:10:10:00/,/22/Apr/2016:10:11:00/p' access.log|awk '{print $1,$6}'|sort|uniq -c |sort -nr|more

    查看某个时间点的

    7.查看服务器并发的

    [root@hemlock php-fpm]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

    [root@hemlock ~]#  netstat -ant | grep ESTABLISHED|more

    查看连接从哪里来

    8.Mongo查看单个IP信息的语法

    . db.access20160417.find({"remote_addr" : "222.64.69.202"})

    10.在10.1.0.9查看这个IP有没登陆网站

    mysql> SELECT * from garmentoffice.login_log where ip_address='5.9.98.178';

    11显示主要信息

     

    11[root@hemlock nginx]# grep "120.236" access.log |awk '{print $1,$5,$6,$8,$9,$10,$11,$12,$13,$14,$15}' |sort |uniq -c |sort -nr|more

       1287 120.236.141.135 "GET /websocket/GetMemberInfo 403 0.002 "http://www.haodingdan.com/company-335380.html?spm=10001" "Mozilla/5.0 (Windows NT 6.1; WOW64)

    显示了ip数和ip  显示了useragent  显示了utl  显示了主页  显示了状态 和显示request_time=0.002  GET是访问的uri  后面url是referer

    12.统计目前访问最多的uri 和ip和其他参数

    awk '{print $1,$5,$6,$8,$9,$10,$11,$12,$13,$14,$15}' access.log |sort |uniq -c |sort -nr|more

    13,知道ip查找来自哪个机房域名的网站

    http://ip.chinaz.com/?qq-pf-to=pcqq.group

    14数据库监控负载的地址

    http://10.1.0.12:8080/render/?width=1133&height=650&target=collectd.quivertree.load.load.shortterm&target=collectd.lime.load.load.shortterm&target=collectd.apple.load.load.shortterm&from=-8hours&fontSize=14

     

    15mongo查看最慢的所有信息中最慢的request_time

    db.access.find({},{"http_x_forwarded_for":1,"request_time":1,"request":1,"time":1}).sort({"request_time":-1}).limit(10);

    注意这里的-1  这里是以request_time排序并且限制出来10条 ,其实mongo最多也就一次出来20条信息

     

    16mongo查看单个IP的信息这里是request.time排序

    db.access.find({http_x_forwarded_for:'220.181.108.160'},{"time":1,_id:0,status:1,request_time:1,http_user_agent:1,request:1,http_x_forwarded_for:1}).sort({request_time:1})

     

     

    17 阿里云工单地址

    https://workorder.console.aliyun.com/console.htm#/ticket/detail/?ticketId=DG6M0A9

     

    18.云监控安装步骤

    https://help.aliyun.com/document_detail/cms/User_Manual/CloudServiceMonitor/CloudMonitorGuide.htm

    选择安装新版云监控插件

     

    19. mongodb数据库删除集合

    db.collection.drop()删除一个集合的命令

    比如 db.access20160511.drop()

     

    20.centos7.1安装计划任务

    yum install crontabs  -y

     

    21 show variables   like  "%gen%";

    搜索mysql的某个配置

     

     

    22 php的平滑重启

    kill  -USR2  `cat /run/php-fpm/php-fpm.pid` 

     

    23  mongo的正则匹配查询只要加一个$regex 就可以查询/sync/check/后面可能还很多。都可以查询得到

    db.access20160622.find({"request":{$regex:"/sync/check"}}).limit(5)

     

    24 cedar每日统计错误代码比如500 404

     nohup php script.php backend/mongo_log/countStatus/access20160603 >> access.log &

     

    25  查询某个字段相关一共有多少条数据

     db.user_inbox_session.find({unread_count:-1}).count()

     

    26 查看访问的request、

     sh check_options.sh  danxin2.haodingdan.com|awk '{print $6,$7,$15}'|sort|uniq -c |sort -nr|more

     

    27阿里云app地址

    https://push.console.aliyun.com/?spm=5176.doc34370.2.2.kJCWpr#/app/deviceStat/?appId=2308580&appKey=23287577&appName=%E5%8D%83%E8%AF%A2&source=

     

    28 mongo的模糊查询

    db.access20160701.find({"request":{$regex:"session_id=9/-1000038/0,0&to=-1000038&user_id=342322"}},{request:1,time:1,_id:0,request_time:1}).sort({request_time:-1})

     

     

    29 查看mongo的索引

    HaodingdanRepSet:PRIMARY> db.user_inbox_session.getIndexes();

    db.purchase_intention.getIndexes();

     

    30 查看mongo表中的一条信息i

    db.user_inbox_session.find().limit(1);

     

    31 如果需要屏蔽改成1 下面是屏蔽的写法,一定指定id ,要不然全部都会被屏蔽   不屏蔽是0 ·执行的时候给老大看下····

     update haodingdan.com_base_info set is_shield = 1 where member_id in( 1212 ,12121,);

     

    32 统计danxin2站有referer_http

    cat  /var/log/nginx/access.log|grep danxin2.haodingdan.com|awk '{print $10}'|sort |uniq -c |sort -nr|more

    说明 $10正好是_这个referer位置

     

    33 统计日志请求大于3就

    sh query_time.sh 08:00 09:00 |grep  /websocket/CheckOnline |awk '{if ($5 > 3) print $1,$2,$3,$4,$5,$6,$7,$8}'

     

    34 查看是否有索引

    show index from tblname;

     

    35  svn拉取之前的版本代码下来,注意我这里只能拉取目录 不能拉取文件  -r后面的1862是指定show log最后一次的版本号

     

    svn  co -r 1862  svn://packages.garmentoffice.org/hdd_modules/business_objects/branches/1.6.0/src/Haodingdan/BusinessModule/

    Python

    #统计PV,UV

    #-*-  coding: UTF-8  -*-
    
    ips = []
    
    with open(r'C:UsersqingheshDesktopview
    xd-frontal-gateway.2017-12-17.log') as f:
    for line in f:
    ip = line.split(',')[1].split(':')[1].strip('"')
    ips.append(ip)
    
    print('PV is {0}'.format(len(ips)))
    print('UV is {0}'.format(len(set(ips))))
    print('The ip is {}'.format(set(ips)))
    
     
    
    #统计httpd code
    
    d = {}
    
    with open('access_log',encoding='utf-8') as f:
    for line in f:
    key = line.split()[8]
    d.setdefault(key,0)
    d[key] += 1
    
    
    sum_request = 0
    err_request = 0
    
    
    for key,val in d.items():
    print(key,val)
    if int(key) >= 400:
    err_request += val
    sum_request+= val
    print(sum_request)
    
    
    print('error rate: {0:.2f}%'.format(err_request * 100.0 / sum_request))

    #统计url

    from collections import Counter
    from collections import Iterator
    
    c = Counter()
    with open('access_log') as f:
    for line in f:
    c[line.split()[6]] += 1
    
    print("Poular resources : {0}".format(c.most_common(3)))
  • 相关阅读:
    从Oracle提供两种cube产品说开
    Sql Server DWBI的几个学习资料
    Unload Oracle data into text file
    初学Java的几个tips
    我常用的Oracle知识点汇总
    benefits by using svn
    如何在windows上使用putty来显示远端linux的桌面
    building commercial website using Microsoft tech stack
    Understand Thread and Lock
    Update google calendar by sunbird
  • 原文地址:https://www.cnblogs.com/qinghe123/p/8134635.html
Copyright © 2020-2023  润新知