• 哈,又一款超级简单的队列(MQ)实现方案来了~


    开源的消息队列已经很多了,但大部分很重,实际环境下,很多可能只是使用到了一点功能而已,杀鸡使用牛刀,着实有些浪费了。很多时候,我们只想要一片绿叶,但它们给了我们整个的春天,很难消化。本着DIR精神, 也琢磨了一个超级简单的队列实现。
    说是超级简单,嗯,绝对是超级简单,队列的存储采用Redis进行持久化存储,采用Netty提供HTTP方式的队列的出/入。Redis的客户端采用的Jedis。然后呢,然后就没了啊。

    一。Redis

    Redis内置订阅发布模型(Publish/Subscribe),其缺陷是,不存储,一旦订阅者断线,将无法接收到消息,再次连接上之后,在断线期间发布者发布的消息都是无法获取到的。只能采用list数组实现,采用rpush/lpop组合命令来实现先进先出的队列模型,当然redis也提供了阻塞版本的blpush/brpush/blpop/brpop等,就看我们实际环境下如何使用了。
    JAVA客户端使用Jedis,提供接口也很丰富。但要注意的是,需要使用连接池,否则在大数量的情况下,有可能jedis的连接不够用。
    private static JedisPool pool;
     static {
      ResourceBundle bundle = ResourceBundle.getBundle("redis");
      if (bundle == null) {
       throw new IllegalArgumentException(
         "cannot find the SignVerProp.properties");
      }
      JedisPoolConfig config = new JedisPoolConfig();
      config.setMaxActive(Integer.valueOf(bundle
        .getString("redis.pool.maxActive")));
      config.setMaxIdle(Integer.valueOf(bundle
        .getString("redis.pool.maxIdle")));
      config.setMaxWait(Integer.valueOf(bundle
        .getString("redis.pool.maxWait")));
      pool = new JedisPool(config, bundle.getString("redis.server"),
        Integer.valueOf(bundle.getString("redis.port")));
     }
    

    二。Netty

    很成熟的NIO框架,用它来提供HTTP方式的队列的出入。嗯,目前只提供HTTP方式的入队列,出队列等。
    HTTP形式为:http://服务器名称:端口/队列操作原语/队列名称?msg=消息内容 
    队列操作原语,只有get和put;get为出队列,put为入队列。
    返回为json: {s:0, m:'错误消息/消息内容'} 
    s为success的缩写,值为0,意味着false,1对应成功。 m为message的缩写,错误消息/消息内容,添加消息时,会返回插入消息对应的数目 默认采用UTF-8。
    入队列:http://localhost:8080/put/demo?msg=消息内容 
    出队列:http://localhost:8080/get/demo HTTP方式,特别适合局域网之间,消息数据的推送。
     

    三。入口解读

    采用了QueueDaemon封装了netty代码: 调用很简单,程序启动的入口: 而HTTP方式队列请求处理器为HttpRequestHandler:

    四。队列处理器

    这个很简单,直接采用Jedis客户端即可,这里一直占用一个连接,不释放。

    五。ab压力测试

    本机配置:Pentium(R) Dual-Core CPU E5200 2.50GHz,2.00 GB内存,Windows XP系统,redis-2.4.5-win32 32版本(非linux版本),都在一台机器上运行。 插入 512 bytes 文本消息队列: 1781.24 requests/sec
    D:Apache2.2in>ab -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    Benchmarking localhost (be patient)
    Completed 10000 requests
    Completed 20000 requests
    Completed 30000 requests
    Completed 40000 requests
    Completed 50000 requests
    Completed 60000 requests
    Completed 70000 requests
    Completed 80000 requests
    Completed 90000 requests
    Completed 100000 requests
    Finished 100000 requests
    Server Software:
    Server Hostname:        localhost
    Server Port:            8080
    Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Document Length:        12 bytes
    Concurrency Level:      10
    Time taken for tests:   56.141 seconds
    Complete requests:      100000
    Failed requests:        99991
       (Connect: 0, Receive: 0, Length: 99991, Exceptions: 0)
    Write errors:           0
    Total transferred:      8188895 bytes
    HTML transferred:       1588895 bytes
    Requests per second:    1781.24 [#/sec] (mean)
    Time per request:       5.614 [ms] (mean)
    Time per request:       0.561 [ms] (mean, across all concurrent requests)
    Transfer rate:          142.45 [Kbytes/sec] received
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   1.7      0      16
    Processing:     0    5  10.7      0     781
    Waiting:        0    5  10.4      0     766
    Total:          0    6  10.8      0     781
    Percentage of the requests served within a certain time (ms)
      50%      0
      66%     16
      75%     16
      80%     16
      90%     16
      95%     16
      98%     16
      99%     16
     100%    781 (longest request)
    
    插入 512 bytes 文本消息队列(添加Keep-Alive支持): 1875.18 requests/sec
    D:Apache2.2in>ab -k -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    Benchmarking localhost (be patient)
    Completed 10000 requests
    Completed 20000 requests
    Completed 30000 requests
    Completed 40000 requests
    Completed 50000 requests
    Completed 60000 requests
    Completed 70000 requests
    Completed 80000 requests
    Completed 90000 requests
    Completed 100000 requests
    Finished 100000 requests
    Server Software:
    Server Hostname:        localhost
    Server Port:            8080
    Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Document Length:        17 bytes
    Concurrency Level:      10
    Time taken for tests:   53.328 seconds
    Complete requests:      100000
    Failed requests:        0
    Write errors:           0
    Keep-Alive requests:    0
    Total transferred:      8300000 bytes
    HTML transferred:       1700000 bytes
    Requests per second:    1875.18 [#/sec] (mean)
    Time per request:       5.333 [ms] (mean)
    Time per request:       0.533 [ms] (mean, across all concurrent requests)
    Transfer rate:          151.99 [Kbytes/sec] received
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   1.5      0      16
    Processing:     0    5   7.8      0     203
    Waiting:        0    5   7.8      0     203
    Total:          0    5   7.9      0     203
    Percentage of the requests served within a certain time (ms)
      50%      0
      66%      0
      75%     16
      80%     16
      90%     16
      95%     16
      98%     16
      99%     16
     100%    203 (longest request)
    
    获取 512 bytes 消息(With Keep-Alive): 1875.73 requests/sec
    D:Apache2.2in>ab -k -c 10 -n 100000 "http://localhost:8080/get/demo"
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    Benchmarking localhost (be patient)
    Completed 10000 requests
    Completed 20000 requests
    Completed 30000 requests
    Completed 40000 requests
    Completed 50000 requests
    Completed 60000 requests
    Completed 70000 requests
    Completed 80000 requests
    Completed 90000 requests
    Completed 100000 requests
    Finished 100000 requests
    Server Software:
    Server Hostname:        localhost
    Server Port:            8080
    Document Path:          /get/demo
    Document Length:        523 bytes
    Concurrency Level:      10
    Time taken for tests:   53.313 seconds
    Complete requests:      100000
    Failed requests:        0
    Write errors:           0
    Keep-Alive requests:    0
    Total transferred:      58900000 bytes
    HTML transferred:       52300000 bytes
    Requests per second:    1875.73 [#/sec] (mean)
    Time per request:       5.331 [ms] (mean)
    Time per request:       0.533 [ms] (mean, across all concurrent requests)
    Transfer rate:          1078.91 [Kbytes/sec] received
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   1.9      0      16
    Processing:     0    5   7.5      0      94
    Waiting:        0    4   6.9      0      94
    Total:          0    5   7.6      0      94
    Percentage of the requests served within a certain time (ms)
      50%      0
      66%      0
      75%     16
      80%     16
      90%     16
      95%     16
      98%     16
      99%     16
     100%     94 (longest request)

    其它问题

    1. 暂时对安全/授权没有支持,这个其实很容易实现
    2. 队列处理器很简单,直接使用jedis即可
    3. 对队列数据进行分片(sharding),写一个QueueService实现即可
    4. 对Redis的内存持久化不放心,采用diskstore存储模式好了,在Redis中配置,不涉及到程序
    5. 对队列分布式存储,写一个QueueService实现,其它可不用变化
    6. 不适合严格意义上的订阅/发布模型,这里适合多个发布者/单个订阅者环境
    7. HTTP请求返回内容为json格式,xml格式,暂时不需要
    8. 局域网环境下,系统之间进行消息的推送/通知,
    项目下载地址:http://code.google.com/p/nettyqueue/ 参考资料:
    1. httpsqs

    http://www.blogjava.net/yongboy/archive/2012/03/20/372308.html

  • 相关阅读:
    Android 数据库
    HarmonyOS三方件开发指南(12)——cropper图片裁剪
    从微信小程序到鸿蒙js开发【15】——JS调用Java
    鸿蒙的js开发部模式16:鸿蒙布局Grid网格布局的应用一
    从微信小程序到鸿蒙js开发【14】——自定义组件
    鸿蒙js开发7 鸿蒙分组列表和弹出menu菜单
    HarmonyOS三方件开发指南(11)——Updownfile
    鸿蒙Java开发模式11:鸿蒙图片裁剪功能的实现
    从微信小程序到鸿蒙js开发【13】——list加载更多&回到顶部
    从微信小程序到鸿蒙js开发【12】——storage缓存&自动登录
  • 原文地址:https://www.cnblogs.com/softidea/p/5568904.html
Copyright © 2020-2023  润新知