合理解决秒级并发100次任务文件监控

个人理解：

首先要理解并发的概念，字面意思就是并行发生。当有大量事务需要处理的时候，就产生的并发，当只有一个处理器的core的时候，事实上同一个时刻只有一件事务可以处理，即便是超线程技术，同时也只能处理一件事务，当然cpu的频率已经非常高了，比如一个核心的主频是1GHz，也就意味着每秒钟开关1,073,741,824（1*1024*1204*1024）次，合理利用时间间隔，也可以给人是并行的错觉。

遇到问题：

这次是遇到了一个任务，主要内容是：监控文件目录，将发生变化的文件，及时传输到指定远程服务器，经服务器计算该文件，返回该文件变动属性。而且不同的文件，需要发往不同的服务器。

考虑因素：

1.服务的承载能力。

2.文件变动的速率和个数。

3.本地处理器的计算能力，cpu最多可以让出多少计算能力。

4.选用什么技术进行传输

5.选用什么技术进行监控文件

6.选用什么框架进行并行

设计方法：

在linux下，以阻塞方式监控文件描述符的属性变化，将写完毕的文件，经过判断文件名筛选掉一部分不需要的文件，然后把加入执行队列。在另一头开启一个进程监控该队列是否消息（该队列是否需要阻塞？），如果有消息就立即转发，利用根据文件的属性发往不同的服务器，传输方式是长连接tcp上的http消息，但是http有一个弊端，需要等待响应的到来（具体不是很清楚是否一定要等待响应，但是我做的请求都是需要等待响应一个  200 ok，比较迷），这就大大降低的传输能力，但是没办法，因为这里的响应结果是有用的数据，一来可以判断是否成功发送，二来可以得到文件属性变动的数据。为了增加传输能力，只能在这一步上做出牺牲，其实我仔细想想，服务器端可以累计再响应，单个就响应返回比较费力不讨好。这里需要做的就是在响应完毕时立即进行下一次数据的发送。同时开启100线程直接可以让操作系统卡死，而且http的传输能力不足以这么做，这里采用5个线程，考虑资源问题，在不需要的可以将线程挂起，而不是销毁，这就有点线程池的感觉了，而且可以做到在下一次消息到来的时候，1号线程恰巧执行上次传输任务完毕，那么可以继续使用该线程，而不是在线程池里再拿2号线程，产生多余的消耗。当没有空余线程的时候，消息应该在消息队列中阻塞，这里设计一个可以承载100个消息的队列。

小结：

1.inotify+epoll+消息队列+线程池+http+mysql（主要任务，也是方便自己理解这些技术）

2.inotify+redis+http(备用)

3.inotify+redis+udp(服务器可以改的话)

 1.inotify.c

//inotify test
//

#include <errno.h>
#include <poll.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/inotify.h>
#include <unistd.h>
#include <pthread.h>
#include "httppost.h"
#include "queue.h"
/* Read all available inotify events from the file descriptor 'fd'.     
wd is the table of watch descriptors for the directories in argv.       
argc is the length of wd and argv.                                      
argv is the list of watched directories.                                
Entry 0 of wd and argv is unused.*/

#define EVENT_NUM 12

char *event_str[EVENT_NUM] = 
{
    "IN_ACCESS",
    "IN_MODIFY",
    "IN_ATTRIB",
    "IN_CLOSE_WRITE",
    "IN_CLOSE_NOWRITE",
    "IN_OPEN",
    "IN_MOVED_FROM",
    "IN_MOVED_TO",
    "IN_CREATE",
    "IN_DELETE",
    "IN_DELETE_SELF",
    "IN_MOVE_SELF"
};

void recv_msg_handle(void* p)
{
    struct mymsg msgbuf;
    key_t key;
    int msgid;

    key=get_key();
    if(key<0)
    {
        perror("get key error");
        exit(1);
    }
    
    msgid=msgget(key,IPC_CREAT|0644);
    if(msgid<0)
    {
        perror("msgget error");
        exit(1);
    }

    while(1)
    {
        const char * filename = receive_msg(msgid,&msgbuf,0);
        if(filename)
        {
           picture_http_post(filename); 
        }
        sleep(0);
    }

    del_msg(msgid);
}

static void handle_events(int fd, int *wd, int argc, char* argv[])
{
    /* Some systems cannot read integer variables if they are not
    properly aligned. On other systems, incorrect alignment may
    decrease performance. Hence, the buffer used for reading from
    the inotify file descriptor should have the same alignment as
    struct inotify_event. */

    char buf[4096] __attribute__ ((aligned(__alignof__(struct inotify_event))));
    const struct inotify_event *event;
    int i;
    ssize_t len;
    char *ptr;
    int ret;
    struct mymsg msgbuf;
    key_t key;
    int msgid;

    key=get_key();
    if(key<0)
    {
        perror("get key error");
        exit(1);
    }
    
    msgid=msgget(key,IPC_CREAT|0644);
    if(msgid<0)
    {
        perror("msgget error");
        exit(1);
    }

    /* Loop while events can be read from inotify file descriptor. */

    for (;;) {

        /* Read some events. */

        len = read(fd, buf, sizeof buf);
        if (len == -1 && errno != EAGAIN) {
            //perror("read");
            //exit(EXIT_FAILURE);
        }
        /* If the nonblocking read() found no events to read, then
        it returns -1 with errno set to EAGAIN. In that case,
        we exit the loop. */

        if (len <= 0)
            break;

        /* Loop over all events in the buffer */

        for (ptr = buf; ptr < buf + len; ptr += sizeof(struct inotify_event) + event->len) 
        {
            event = (const struct inotify_event *) ptr;

            /* Print event type */
            for (int i = 0; i < EVENT_NUM; i++)
            {
                if ((event->mask >> i) & 1)
                {
                    if (event->len > 0)
                    {
                        printf("%s ----- %s 
", event->name, event_str[i]);
                    }
                    else
                    {
                        printf("directory --- %s
", event_str[i]);
                    }
                }
            }

            if (event->mask & IN_OPEN)
                printf("IN_OPEN: ");
            if (event->mask & IN_CLOSE_NOWRITE)
                printf("IN_CLOSE_NOWRITE: ");
            if (event->mask & IN_CLOSE_WRITE){
                printf("IN_CLOSE_WRITE: ");
                int ret = send_msg(msgid,msgbuf,(const char*)event->name);
                if(-1 == ret)
                {
                    printf("send_msg send error!
");
                }
            }
                

            /* Print the name of the watched directory */
            for (i = 1; i < argc; ++i) {
                if (wd[i] == event->wd) {        
                    printf("%s/", argv[i]);
                    break;
                }
            }

            /* Print the name of the file */

            if (event->len)
                printf("%s", event->name);

            /* Print type of filesystem object */

            if (event->mask & IN_ISDIR)
                printf(" [directory]
");
            else
                printf(" [file]
");
        }
    }
    sleep(0);
}

int main(int argc, char* argv[])
{
    char buf;
    int fd, i, poll_num;
    int *wd;
    nfds_t nfds;
    struct pollfd fds[2];
    pthread_t thread;

    //需要再同级目录下新建三个文件
    freopen("./filein.log","r",stdin);
    freopen("./fileout.log","w",stdout);
    freopen("./fileerr.log","w",stderr);

    
    if (argc < 2) {
        printf("Usage: %s PATH [PATH ...]
", argv[0]);
        exit(EXIT_FAILURE);
    }

    printf("Press ENTER key to terminate.
");

    /* Create the file descriptor for accessing the inotify API */

    fd = inotify_init1(IN_NONBLOCK);//
    if (fd == -1) {
        printf("inotify_init1 error");
        exit(EXIT_FAILURE);
    }

    /* Allocate memory for watch descriptors */

    wd = calloc(argc, sizeof(int));
    if (wd == NULL) {
        printf("calloc error");
        exit(EXIT_FAILURE);
    }

    /* Mark directories for events
    - file was opened
    - file was closed */

    for (i = 1; i < argc; i++) {
        wd[i] = inotify_add_watch(fd, argv[i], IN_ALL_EVENTS);//添加监视对象
        if (wd[i] == -1) {
            printf( "Cannot watch '%s'
", argv[i]);
            printf("inotify_add_watch error");
            exit(EXIT_FAILURE);
        }
    }

    /* Prepare for polling */

    nfds = 2;

    /* Console input */

    fds[0].fd = STDIN_FILENO;//一般指输入设备的文件描述符
    fds[0].events = POLLIN;

    /* Inotify input */

    fds[1].fd = fd;
    fds[1].events = POLLIN;


    pthread_create(&thread, NULL, (void *)&recv_msg_handle, (void *)NULL);
    /* Wait for events and/or terminal input */

    printf("Listening for events.
");
    while (1) {
        poll_num = poll(fds, nfds,-1);//把当前的文件指针挂到等待队列
        if (poll_num == -1) {
            if (errno == EINTR)
                continue;
            printf("poll error");
            exit(EXIT_FAILURE);
        }

        if (poll_num > 0) {

            if (fds[0].revents & POLLIN) {

                /* Console input is available. Empty stdin and quit */

                while (read(STDIN_FILENO, &buf, 1) > 0 && buf != '
')
                    continue;
                //break;
            }

            if (fds[1].revents & POLLIN) {

                /* Inotify events are available */

                handle_events(fd, wd, argc, argv);
            }
        }
        sleep(0);
    }

    printf("Listening for events stopped.
");

    /* Close inotify file descriptor */
    close(fd);
    free(wd);
    exit(EXIT_SUCCESS);
}

2.httppost.h

#ifndef __HTTP_POST__
#define __HTTP_POST__

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <string.h>
#include <sys/time.h>
#include <sys/select.h>
#include <fcntl.h>
#include <errno.h>

#define ADDRESS_NUMBER 3

extern const char* ip[ADDRESS_NUMBER];
extern unsigned int port[ADDRESS_NUMBER];

typedef struct ip_port
{
    char ip[256];
    int port;
}ip_port;


extern int picture_http_post(const char *argv);

#endif

3.httppost.c

#include "httppost.h"

ip_port addr[ADDRESS_NUMBER];
const char* ip[]={"192.168.100.100","192.168.100.107","192.168.100.108"};
unsigned int port[]={15002,15003,15004};

static const char upload_head[] =
    "POST /service/run_json_decs HTTP/1.1
"
    "Host: %s:%d
"
    "Connection: keep-alive
"
    "Content-Type: multipart/form-data; boundary=%s
"
    "Content-Length: %d

"
    "--%s
"
    "Content-Disposition: form-data; name="image"; filename="%s"
"
    "Content-Type: application/octet-stream;chartset=UTF-8

";
    
static const char upload_request[] = 
    "--%s
"
    "Content-Disposition: form-data; name="file"; filename="%s"
"
    "Content-Type: application/octet-stream;chartset=UTF-8

";

static int connect_server(const char *severip,const int port)
{
    int sock = -1;
    struct sockaddr_in addr;
    
    if(severip == NULL || port <= 0)
        return -1;
        
    memset(&addr,0,sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_port = htons(port);
    inet_pton(AF_INET,severip,&addr.sin_addr.s_addr);
    
    sock = socket(AF_INET,SOCK_STREAM,0);
    if(-1 == sock)
        return -1;

    if(connect(sock,(struct sockaddr*)&addr,sizeof(addr)) < 0){
        close(sock);
        return -1;
    }
        
        return sock;    
    
}

ip_port select_addr(ip_port *addr,char *filename)
{
    //addr copy
    for (int i = 0; i < ADDRESS_NUMBER; i++)
    {
        strncpy(addr[i].ip,ip[i],256);
        addr[i].port = port[i];
    }
    
    return addr[0];
}

int picture_http_post(const char *argv)
{
    char filename[256] = {0};
    int filejudge = 0;
    int sock = -1;
    char send_date[4096] = {0};
    char send_end[128] = {0};
    char send_request[2048] = {0};
    FILE *fp = NULL;
    char boundary[64] = {0};
    int ret = -1,already = -1,ContentLength = -1;
    long long int timestamp;
    struct timeval tv;
    


    strncpy(filename,argv,256);

    if(access(filename,F_OK) != 0){
        printf("file %s not exsit!
",filename);
        return -1;
    }
//Resolves whether the filename ends in JPG
    filejudge = strlen(filename);
    if(filename[filejudge-1]!='g'||filename[filejudge-2]!='p'||filename[filejudge-3]!='j'){
        printf("file %s is not jpg!
",filename);
        return -1;
    }
//地址选择
    ip_port transmitaddr = select_addr(addr,filename);
    if(transmitaddr.port == 0){
        printf("address error!
");
    }

//Connect to server
    sock = connect_server(transmitaddr.ip,transmitaddr.port);
    if(sock < 0 ){
        printf("connect server error!
");
        return -1;
    }
    
//Open the file to upload and get the file size for calculating the content-Length size
    fp = fopen(filename,"rb");
    if(fp == NULL){
        printf("open file %s error!
",filename);
        close(sock);
        return -1;
    }
    
    fseek(fp,0,SEEK_END);
    ContentLength = ftell(fp);
    rewind(fp);
    
//Obtain the time stamp of millisecond level to use for the value of the boundary
    gettimeofday(&tv,NULL);
    timestamp = (long long int)tv.tv_sec * 1000 + tv.tv_usec;
    snprintf(boundary,64,"---------------------------%lld",timestamp);
    
//The content-length size also includes the description of the uploaded file, starting boundary and ending boundary
    ContentLength += snprintf(send_request,2048,upload_request,boundary,filename);
    ContentLength += snprintf(send_end,2048,"
--%s--
",boundary);
    
//A description of the HTTP headers sent and the uploaded files
    ret = snprintf(send_date,4096,upload_head,transmitaddr.ip,transmitaddr.port,boundary,ContentLength,boundary,filename);

    if(send(sock,send_date,ret,0) != ret){
        printf("send head error!
");
        close(sock);
        return -1;
    }
    
//The loop reads the file and sends it to the server until the file is finished
    clearerr(fp);
    for(;;){
            memset(send_date,0,sizeof(send_date));
            ret = fread(send_date,1,4096,fp);
            if(ret != 4096){
                    if(!ferror(fp)){
                        if(send(sock,send_date,ret,0) != ret){
                                printf("send the end date error!
");
                                close(sock);
                                fclose(fp);
                                return -1;
                        }
                        fclose(fp);
                        break;
                    }
                    else{
                        printf("read file error!
");
                        close(sock);
                        fclose(fp);
                        return -1;
                    }
                        
            }
            
            if(send(sock,send_date,4096,0) != 4096){
                    printf("send date error
");
                    close(sock);
                    fclose(fp);
                    return -1;
            }
        sleep(0);    
    }

// Send the final boundary closing file for uploading.
SEND_END:        
    memset(send_date,0,sizeof(send_date));
    ret = snprintf(send_date,4096,"
--%s--
",boundary);
    if(send(sock,send_date,ret,0) != ret){
            close(sock);
            return -1;
    }

    printf("send to server end date:%s
",send_date);

//Receives the return value to determine if the upload was successful
    memset(send_date,0,sizeof(send_date));
    if(recv(sock,send_date,4096,0) < 0)
        printf("recv error!
");
        
    printf("recv:%s
",send_date);
    close(sock);
    
    return 0;
}

4.queue.h

#ifndef __QUEUE__
#define __QUEUE__

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ipc.h>
#include <sys/msg.h>
#include <sys/types.h>
#include <error.h>

#define BUF_SIZE 256

struct mymsg
{
    long mtype;
    char mtext[BUF_SIZE];
};

extern int send_msg(int msgid,struct mymsg msgbuf,const char *filename);
extern char* receive_msg(int msgid,struct mymsg *qbuf,long type);
extern void del_msg(int msgid);
extern int get_key();

//执行接收任务
extern void recv_msg_handle(void* p);

#endif

5.queue.c

#include "queue.h"

int get_key()
{
    int key;
    key=ftok(".",'s');
}

int send_msg(int msgid,struct mymsg msgbuf,const char *file_name)
{    
    printf("enter the msg text
");
    //const char *file_name="test.jpg";
    strncpy(msgbuf.mtext, file_name, BUF_SIZE);
    msgbuf.mtype=1;//消息的类型
    //int msgsnd(int msqid, const void *msgp, size_t msgsz, int msgflg);
    if(msgsnd(msgid,(void *)&msgbuf,BUF_SIZE,0)==-1)
    {
        perror("send msg error");
        return -1;
    }
    return 0;
}

char* receive_msg(int msgid,struct mymsg *qbuf,long type)
{
    //ssize_t msgrcv(int msqid, void *msgp, size_t msgsz, long msgtyp,int msgflg);
    if(msgrcv(msgid,qbuf, BUF_SIZE,type,0)==-1)
    {
        printf("rcv msg error
");
        return NULL;
    }
    else printf("Type:%ld,Text:%s
",qbuf->mtype,qbuf->mtext);
    return (char*)qbuf->mtext;
}
 
void del_msg(int msgid)
{
    if(msgctl(msgid,IPC_RMID,0)==-1)
    {
        perror("rm msg error");
        exit(1);
    }
}

6.readme

1. install inotify-tools
   sudo apt-get install inotify-tools

2.工作目录下有fileerr.log,filein.log,fileout.log三个文件

3.可以在httppost.c中设置多路ip和port。

4. ./build_sh


5. sudo ./monitoring 目录 & 


6.设置开机自启动
    1.需要在automatic.sh中，前一个PATH是automatic.sh的绝对路径，第二个PATH是工作路径

    2.设置权限 sudo  chmod  777  automatic.sh
    3.移动文件 sudo  mv  automatic.sh  /etc/init.d/
    4.设置自启 cd /etc/init.d/ && sudo update-rc.d automatic.sh defaults 75
    5.关闭自启 sudo update-rc.d -f automatic.sh remove

7.build.sh

#!/bin/bash

gcc httppost.c httppost.h queue.c queue.h inotify.c -lpthread -o monitoring

8.automatic.sh

#!/bin/bash
 
### BEGIN INIT INFO
# Provides:     automatic
# Required-Start:  $remote_fs $syslog
# Required-Stop:   $remote_fs $syslog
# Default-Start:   2 3 4 5
# Default-Stop:   0 1 6
# Short-Description: start automatic
# Description:    start automatic
### END INIT INFO

cd /path && (./monitoring ./ &) #需要根据具体路径修改

exit 0

最后抓包看一下：

nice，ok

结语：

最终采用了消息队列，并没有用到线程池，因为http上报后响应需要浪费挺多时间，当然，线程池我也实现了一下，逻辑功能上加了可以转发多路ip，http上报采用的content type是multipart/form-data，每一次的请求完毕都会主动关闭socket，当下一次http上报的任务来临时重新开启，这个程序还可以加上一段频繁上报的时候，不关闭socket，因为我才用的连接方式时keep alive，http时基于tcp直接实现的，并没有调用库。

个人总结，有疑问可以联系我。