• 异步多线程C/S框架gko_pool


    曾经在上家公司做了一个内网P2P的传输工具名曰gingko 当时太菜了,网络部分写的很差,各种阻塞式网络通信 后来离职把memcached的代码研究了一下,感觉里面的网络模型非常先进 我就把memcached的网络部分抽取出来和gingko做了结合,写成了一个网络框架叫gko_pool 大致特性是:

    • 完全异步化(包括DNS解析)
    • 可以listen一个端口当server,也可以主动发起连接写client
    • worker是对等的多线程
    • 内置一个简单的“内存池”

    看代码发现memcached状态机有个逻辑上的bug 顺手给memcached提了个patch 后来这事就被淡忘了,过了两年被维护者发现合到master上了,维护者是这么说的:

    Wow, two years old... and it looks correct to me. If that update_event fails the connection might zombie. It's very hard for that to fail and it's been that way forever. Pushed a commit for the next release.

    言归正传,可以毫不夸张的说memecached是多线程异步网络编程的一个丰碑(nginx是多进程异步网络编程的一个丰碑) 我就把memcached的网络部分抽取出来写成了一个网络框架叫gko_pool https://github.com/auxten/gko_pool

     

    举个栗子:http刷票器,在我的笔记本虚机上也能达到300KQPS,控制了一下速度,一不小心就把sina的服务器打挂了

    /*
     *
     *  Created on: May 18, 2012
     *      Author: auxten
     */
    #include "gko.h"
    #include <sys/time.h>
    /// gingko global stuff
    s_gingko_global_t gko;
    
    pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
    int cnt = 3;
    int counter = 0;
    void report_result(void * c, const char * msg)
    {
        GKOLOG(NOTICE, "%s", msg);
        pthread_mutex_lock(&lock);
        if (++counter == cnt)
        {
            printf("finished
    ");
            exit(0);
        }
        pthread_mutex_unlock(&lock);
    }
    
    int main(int argc, char** argv)
    {
        char cmd[2048];
    
        gko.opt.to_debug = 0;
        gko.ready_to_serv = 1;
        gko.sig_flag = 0;
        gko.opt.worker_thread = 8;
        gko.opt.connlimit = SERV_POOL_SIZE;
    //    gko.opt.to_debug = 1;
        gko_pool * gingko = gko_pool::getInstance();
        gingko->setPort(-1);
        gingko->setOption(&gko.opt);
    //    gingko->setProcessHandler(conn_send_data);
        gingko->setReportHandler(report_result);
        gingko->gko_run();
    
        char ua[56][256] =
        {
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
        "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)",
        "Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (Linux; U; Android 2.2; fr-fr; Desire_A8181 Build/FRF91) App3leWebKit/53.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0",
        "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.0) Opera 7.02 Bork-edition [en]",
        "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.02",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 3.5.30729)",
        "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)",
        "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal 6.2)",
        "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 5.8 (build 4157); .NET CLR 2.0.50727; AskTbPTV/5.11.3.15590)",
        "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)",
        "Mozilla/5.0 (Windows NT 6.1; rv:2.0b7pre) Gecko/20100921 Firefox/4.0b7pre",
        "Mozilla/5.0 (Windows NT 5.1; rv:5.0.1) Gecko/20100101 Firefox/5.0.1",
        "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4",
        "Opera/9.80 (Windows NT 5.1; U; en) Presto/2.10.289 Version/12.01",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
        "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.92 Safari/537.4",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1",
        "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4",
        "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4",
        "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:15.0) Gecko/20100101 Firefox/15.0.1",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1",
        };
    
        char srv[3][32] =
        {
            "202.108.43.189",
            "202.106.182.237",
            "home"
        };
    
        int i = cnt;
        while (i--)
        {
            struct timeval tv;
            gettimeofday(&tv, NULL);
            long t = tv.tv_usec;
            sprintf(cmd,
                    "GET /Poll.php?project_id=5168&id=49 HTTP/1.%ld
    "
    //                "GET /Poll.php?project_id=5168&id=80 HTTP/1.%ld
    "
                            "Host: hi.video.sina.com.cn
    "
                            "User-Agent: %s
    "
                            "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    "
                            "Accept-Language: zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
    "
                            "DNT: 1
    "
                            "Client-IP: %ld.%ld.%ld.%ld
    "
                            "Connection: keep-alive
    "
                            "Referer: http://hi.video.sina.com.cn/you/2012banjiti/ph.php?dpc=1
    
    "
                    , t % 2,ua[t % 56], 122 + t % 20, 121 + (i + t) % 128, 24 + (i + t) % 135, 24 + (i + t) % 213);
            gingko->make_active_connect(srv[i%3], 80, strlen(cmd), cmd, -1, -1, 0, 10);
    //        gingko->make_active_connect("61.30.127.2", 80, strlen(cmd), cmd, -1, -1, 0, 10);
        }
        sleep(10); //由于发请求时异步的,这里也没去处理response,sleep秒再退出
        return 0;
    }
    
    

    我为了刷票刷的真实,还写了这么多UserAgent,刷票界的良心啊,哈哈哈 更多例子,比如:一个自定义协议的client、server等请移步github 这套库的稳定性大家不用质疑,已经用在了我现在公司的上万台服务器的很多网络Daemon上

     
  • 相关阅读:
    表格行内编辑增删改查
    rpc远程调用开发
    gdb调试程序
    Makefile文件编写
    CentOS6.6源码编译升级GCC至4.8.2
    linux 下C语言编程库文件处理与Makefile编写
    laravel队列-让守护进程处理耗时任务
    回顾javase点滴
    服务端生成word并压缩打包下载
    pytest框架插件(用例依赖、多重校验、执行顺序、失败重跑、重复执行、标记)
  • 原文地址:https://www.cnblogs.com/51reboot/p/4005552.html
Copyright © 2020-2023  润新知