RabbitMQ队列

RabbitMQ队列　　

rabbitmq 消息队列
解耦：一个程序间 把两个耦合度降低

异步：天生解决耦合
优点：解决排队问题 
缺点：不能保证任务被及时的执行
应用场景：去哪儿网 12306网站

同步
优点：保证任务及时执行
缺点：不能解决排队问题,导致时间被浪费

大并发 Web Linux上 近几年使用的是nginx 内部epoll异步 承载10000-20000并发
pv=page visit 页面访问量 每天上亿 10server web cluster集群
uv=user visit 用户访问量 每天600多万
qps=每秒查询率QPS是对一个特定的查询服务器在规定时间内所处理流量多少的衡量标准。

队列的作用
1、存储消息、数据
2、保证消息顺序
3、保证数据的交付

python消息队列中：
只能同个进程多个线程访问使用

为什么使用rabbitmq instead of python queue
python消息队列中：
只能同个进程多个线程访问使用，且不能跨进程

安装

　　centos  　　http://www.rabbitmq.com/install-rpm.html 
　　windows 　 http://www.rabbitmq.com/install-windows.html 
　　mac 　　　 http://www.rabbitmq.com/install-standalone-mac.html

安装python rabbitMQ module 

pip install pika
or
easy_install pika
or
源码
  
https://pypi.python.org/pypi/pika

Centos 安装远程连接示例：

#Centos7 安装
 
#注意/etc/hosts文件 ip和主机名对应
wget https://github.com/rabbitmq/rabbitmq-server/releases/download/rabbitmq_v3_6_10/rabbitmq-server-3.6.10-1.el7.noarch.rpm
yum install epel-release -y
yum install rabbitmq-server-3.6.10-1.el7.noarch.rpm
rabbitmq-plugins enable rabbitmq_management
cp /usr/share/doc/rabbitmq-server-3.6.10/rabbitmq.config.example /etc/rabbitmq/rabbitmq.config
systemctl restart rabbitmq-server
systemctl status rabbitmq-server
 
#创建用户 授权
rabbitmqctl  add_user xyp 123456
rabbitmqctl set_permissions -p / xyp ".*" ".*" ".*"

实现最简单的队列通信

二. 事例

a. 服务端和客户端一对一 

#查看队列
    # rabbitmqctl list_queues

#客户端再次申明队列是因为客户端要清楚去哪里取数据
    channel.queue_declare(queue='hello')

 

sender.py

import pika
# 授权连接
# credentials = pika.PlainCredentials('xyp', '123456')  #授权的账号xyp，密码123456
# parameters = pika.ConnectionParameters(host='192.168.11.106',credentials=credentials)     #建立socket
# 本地连接
parameters = pika.ConnectionParameters(host='localhost')
connection = pika.BlockingConnection(parameters)
channel = connection.channel() #队列连接通道  #创建rabbitmq协议通道
#声明queue
channel.queue_declare(queue='hello')    #通过通道生成一个队列
# n RabbitMQ a message can never be sent directly to the queue, it always needs to go through an exchange.
channel.basic_publish(exchange='',
                      routing_key='hello',        #队列
                      body='Hello World!')      # 内容
print(" [x] Sent 'Hello World!'")
connection.close()

receive.py

import pika
#授权的账号xyp
# credentials = pika.PlainCredentials('xyp', '123456')
# parameters = pika.ConnectionParameters(host='192.168.11.106 ',credentials=credentials)
# 本地连接
parameters = pika.ConnectionParameters(host='localhost')
connection = pika.BlockingConnection(parameters)    # 建立socket
channel = connection.channel() #队列连接通道
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
    print(" [x] Received %r" % ch, method, properties, body)
    # ch ，上面channel = connection.channel()得到的对象
    #method, 除了服务端本身的数据，还带一些参数
    # properties,属性
    # body，bytes数据

channel.basic_consume(callback, #取到消息后，调用callback 函数
                      queue='hello',
                      no_ack=True)

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming() #阻塞模式

运行接收端receive.py，打印：

[*] Waiting for messages. To exit press CTRL+C

然后运行发送端sender.py，打印：

[x] Sent 'Hello World!'

发送端sender.py运行后，接收端receive.py接收到消息，打印：

[x] Received <pika.adapters.blocking_connection.BlockingChannel object at 0x0000000002D6B550> <Basic.Deliver(['consumer_tag=ctag1.81443a42050f4d249478c4011e2fc086', 'delivery_tag=1', 'exchange=', 'redelivered=False', 'routing_key=hello'])> <BasicProperties> b'Hello World!'

以上内容基于本地连接rabbitmq server，远程连接rabbitmq server的话，需要先配置权限

首先在rabbitmq server上创建一个用户

sudo rabbitmqctl  add_user xyp 123456　

同时还要配置权限，允许从外面访问

sudo rabbitmqctl set_permissions -p / xyp ".*" ".*" ".*"

set_permissions [-p vhost] {user} {conf} {write} {read}

vhost

The name of the virtual host to which to grant the user access, defaulting to /.

user

The name of the user to grant access to the specified virtual host.

conf

A regular expression matching resource names for which the user is granted configure permissions.

write

A regular expression matching resource names for which the user is granted write permissions.

read

A regular expression matching resource names for which the user is granted read permissions.

客户端连接的时候需要配置认证参数　

credentials = pika.PlainCredentials('xyp', '123456')
 
 
connection = pika.BlockingConnection(pika.ConnectionParameters(
    '10.211.55.5',5672,'/',credentials))
channel = connection.channel()

　　

b. 模拟客户端中断 观察服务端队列的数据会不会返回(不会)

 　正常模式：

　　　先启动消息生产者send.py，然后再分别启动2个消费者receive.py，通过生产者send.py多发送几条消息，你会发现，这几条消息会被依次分配到各个消费者receive.py身上。

　下面模拟客户端中断模式：

#- 开启一个服务端,两个客户端
#- 服务端向队列中存放一个值,一客户端从队列中取到数据,在睡20秒期间中断,表示出错,它不会报告给服务端
#- 这时队列中为零,另一客户端也不会取到值
# no_ack=True 表示客户端处理完了不需要向服务端确认消息

import pika

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('192.168.11.106',credentials=credentials))  #建立socket

channel = connection.channel()            #创建rabbitmq协议通道

channel.queue_declare(queue='hello')      #通过通道生成一个队列

channel.basic_publish(exchange='',
                      routing_key='hello',      #队列
                      body='Hello World!')      #内容
print(" [x] Sent 'Hello World!'")
connection.close()

import pika
import time

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('192.168.11.106',credentials=credentials))  #建立socket

channel = connection.channel()


channel.queue_declare(queue='hello')


def callback(ch, method, properties, body):
    print("received msg...start process",body)
    time.sleep(10)
    print("end process...")


channel.basic_consume(callback,
                      queue='hello',
                      no_ack=True)

print(' Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

c. 模拟客户端中断 观察服务端队列的数据会不会返回(会) 

#客户端需要向服务端确认
  ch.basic_ack(delivery_tag=method.delivery_tag)
#- 不加 no_ack=True参数
#- 开启一个服务端,两个客户端
#- 服务端向队列中存放一个值,一客户端从队列中取到数据,在睡20秒期间中断,表示出错,它会报给服务端,服务端队列还有值
#- 这时启动另一客户端还可以取到值

import pika
import sys
import time


credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()                 #创建rabbitmq协议通道
channel.queue_declare(queue='task_queue')      #通过通道生成一个队列


#把脚本的参数 或者 Hello World时间传给客户端
message = ' '.join(sys.argv[1:]) or "Hello World! %s" % time.time()

channel.basic_publish(exchange='',
                      routing_key='task_queue',
                      body=message
                      )
print("  Send %r" % message)

import pika
import time

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()


def callback(ch, method, properties, body):
    print(" Received %r" % body)
    time.sleep(20)
    print("Received done")
    print("method.delivery_tag", method.delivery_tag)
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(callback,
                      queue='task_queue',
                      )

print('Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

d. fanout 广播

　　fanout: 所有bind到此exchange的queue都可以接收消息

#服务端：
  - 不需要申明队列
#客户端：
  - 每个客户端都需要申明一个队列,自动设置队列名称,收听广播,当收听完后queue删除
  - 把队列绑定到exchange上
#注意：客户端先打开,服务端再打开,客户端会收到消息

#应用：
  - 微博粉丝在线,博主发消息,粉丝可以收到

import pika
import sys
import time


credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()                             #创建rabbitmq协议通道

channel.exchange_declare(exchange='logs',type='fanout')

message = ' '.join(sys.argv[1:]) or "info: Hello World!"
channel.basic_publish(exchange='logs',
                      routing_key='',
                      body=message)
print(" Send %r" % message)
connection.close()

import pika
import time

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()

channel.exchange_declare(exchange='logs',
                         type='fanout')

result = channel.queue_declare(exclusive=True)  # 不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除
queue_name = result.method.queue

channel.queue_bind(exchange='logs',queue=queue_name)    #把queue绑定到exchange

print(' [*] Waiting for logs. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] %r" % body)


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)

channel.start_consuming()

e. direct 组播：有选择的接收消息

 　　RabbitMQ还支持根据关键字发送，即：队列绑定关键字，发送者将数据根据关键字发送到消息exchange，exchange根据 关键字 判定应该将数据发送至指定队列。

#客户端一设置包含关键词info才可以接收到消息：
    - python3 receive.py info 
#客户端二设置包含关键词error才可以接收信息：
    - python3 receive.py  error
#客户端三设置包含关键词warning才可以接收信息：
    - python3 receive.py  warning
#客户端四设置包含关键词warning error info才可以接收信息
    - python3 receive.py  warning error info
#服务端发送warning：
    - python3 sender.py  warning

　　

import pika
import sys
import time


credentials = pika.PlainCredentials("xyp","123456")                   #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()                  #创建rabbitmq协议通道

channel.exchange_declare(exchange='direct_logs',type='direct')

severity = sys.argv[1] if len(sys.argv) > 1 else 'info'

message = ' '.join(sys.argv[2:]) or 'Hello World!'

channel.basic_publish(exchange='direct_logs',
                      routing_key=severity,
                      body=message)

print(" Send %r:%r" % (severity, message))
connection.close()

import pika
import time
import sys

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()

channel.exchange_declare(exchange='direct_logs',type='direct')

result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue

severities = sys.argv[1:]
if not severities:
    sys.stderr.write("Usage: %s [info] [warning] [error]
" % sys.argv[0])
    sys.exit(1)

for severity in severities:
    channel.queue_bind(exchange='direct_logs',
                       queue=queue_name,
                       routing_key=severity)

print(' [*] Waiting for logs. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
channel.start_consuming()

f. topic 规则传播

　　更细致的消息过滤，利用正则

#客户端一设置匹配*.django可以接收到信息：
    - python3 receive.py *.django 
#客户端二设置匹配mysql.error可以接收到信息：
    - python3 receive.py mysql.error
#客户端三设置匹配 mysql.*可以接收到信息：
    - python3 receive.py mysql.*

#服务端：
    - python3 sender.py  #匹配相应的客户端  

import pika
import time
import sys

credentials = pika.PlainCredentials("xyp","123456")                     #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()

channel.exchange_declare(exchange='topic_logs',type='topic')

result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue

binding_keys = sys.argv[1:]
if not binding_keys:
    print(sys.argv[1:])
    sys.stderr.write("Usage: %s [binding_key]...
" % sys.argv[0])
    sys.exit(1)

for binding_key in binding_keys:
    channel.queue_bind(exchange='topic_logs',
                       queue=queue_name,
                       routing_key=binding_key)

print(' [*] Waiting for logs. To exit press CTRL+C')


def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)

channel.start_consuming()

import pika
import sys
import time


credentials = pika.PlainCredentials("xyp","123456")                   #授权的账号 密码
connection = pika.BlockingConnection(
    pika.ConnectionParameters('172.16.42.128',credentials=credentials))  #建立socket

channel = connection.channel()                  #创建rabbitmq协议通道

channel.exchange_declare(exchange='topic_logs',type='topic')

routing_key = sys.argv[1] if len(sys.argv) > 1 else 'anonymous.info'

message = ' '.join(sys.argv[2:]) or 'Hello World!'
channel.basic_publish(exchange='topic_logs',
                      routing_key=routing_key,
                      body=message)

print(" [x] Sent %r:%r" % (routing_key, message))

e. RPC(Remote procedure call)远程过程调用

receive.py
1.声明一个队列，作为reply_to返回消息结果的队列
2.发消息到队列，消息里带一个唯一标识符uid和reply_to
3.监听reply_to的队列，直到有结果
sender.py
1.定义fib函数
2.申明接受指令的列名rpc_queue
3.开始监听队列，收到消息后调用fib函数
4.把fib执行结果，发送回客户端指定的reply_to 队列

import subprocess
import pika
import time

parameters = pika.ConnectionParameters('localhost')
connection = pika.BlockingConnection(parameters)

channel = connection.channel()  # 队列连接通道

channel.queue_declare(queue='rpc_queue2')


def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n - 1) + fib(n - 2)


def run_cmd(cmd):
    cmd_obj = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    result = cmd_obj.stdout.read() + cmd_obj.stderr.read()

    return result


def on_request(ch, method, props, body):
    cmd = body.decode("utf-8")

    print(" [.] run (%s)" % cmd)
    response = run_cmd(cmd)

    ch.basic_publish(exchange='',
                     routing_key=props.reply_to,  # 队列
                     properties=pika.BasicProperties(correlation_id=props.correlation_id),
                     body=response)

    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(on_request, queue='rpc_queue2')

print(" [x] Awaiting RPC requests")
channel.start_consuming()

import queue

import pika
import uuid


class CMDRpcClient(object):
    def __init__(self):
        # credentials = pika.PlainCredentials('xyp', '123456')
        # parameters = pika.ConnectionParameters(host='192.168.11.106',credentials=credentials)

        parameters = pika.ConnectionParameters('localhost')
        self.connection = pika.BlockingConnection(parameters)
        self.channel = self.connection.channel()

        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue  # 命令的执行结果的queue

        # 声明要监听callback_queue
        self.channel.basic_consume(self.on_response,
                                   no_ack=True,
                                   queue=self.callback_queue)

    def on_response(self, ch, method, props, body):
        """
        收到服务器端命令结果后执行这个函数
        :param ch:
        :param method:
        :param props:
        :param body:
        :return:
        """
        if self.corr_id == props.correlation_id:
            self.response = body.decode("gbk")  # 把执行结果赋值给Response

    def call(self, n):
        self.response = None
        self.corr_id = str(uuid.uuid4())  # 唯一标识符号
        self.channel.basic_publish(exchange='',
                                   routing_key='rpc_queue2',
                                   properties=pika.BasicProperties(
                                       reply_to=self.callback_queue,
                                       correlation_id=self.corr_id,
                                   ),
                                   body=str(n))

        while self.response is None:
            self.connection.process_data_events()  # 检测监听的队列里有没有新消息，如果有，收，如果没有，返回None
            # 检测有没有要发送的新指令
        return self.response


cmd_rpc = CMDRpcClient()

print(" [x] Requesting fib(30)")
response = cmd_rpc.call('ifconfig')

print(response)