celery

介绍

Celery是由Python开发的一个简单、灵活、可靠的处理大量任务的分发系统，它不仅支持实时处理也支持任务调度。

• user：用户程序，用于告知celery去执行一个任务。
• broker: 存放任务（依赖RabbitMQ或Redis，进行存储）
• worker：执行任务

celery需要rabbitMQ、Redis、Amazon SQS、Zookeeper（测试中） 充当broker来进行消息的接收，并且也支持多个broker和worker来实现高可用和分布式。http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html

应用场景

将响应服务器与处理服务器分开,当处理计算密集型请求时,有work来进行计算,客户端访问响应服务器时,响应服务器根据关键字去获取处理结果,不必由响应服务器进行耗时的计算操作

版本和要求

Celery version 4.0 runs on
        Python ❨2.7, 3.4, 3.5❩
        PyPy ❨5.4, 5.5❩
    This is the last version to support Python 2.7, and from the next version (Celery 5.x) Python 3.5 or newer is required.

    If you’re running an older version of Python, you need to be running an older version of Celery:

        Python 2.6: Celery series 3.1 or earlier.
        Python 2.5: Celery series 3.0 or earlier.
        Python 2.4 was Celery series 2.2 or earlier.

    Celery is a project with minimal funding, so we don’t support Microsoft Windows. Please don’t open any issues related to that platform.

　　ps : win作为开发环境下安装3.1版本,否则会出现关于timer的异常

项目依赖

需要安装rabbitMQ、Redis. 以及python操作他们的包

快速上手

创建work

s1.py

import time
from celery import Celery

app = Celery('tasks', broker='redis://192.168.10.48:6379', backend='redis://192.168.10.48:6379')

# broker     放置发布任务,redis://:password@hostname:port/db_number
# backend 放置任务结果 
@app.task 
def xxxxxx(x, y): 
    time.sleep(10) 
    return x + y

启动work

# 执行 s1.py 创建worker（终端执行命令）：
celery worker -A s1 -l info # -A 文件名

发布一个任务

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from s1 import xxxxxx

# 立即告知celery去执行xxxxxx任务，并传入两个参数
result = xxxxxx.delay(4, 4)
print(result.id)

# 执行此脚本即可将任务发布,会有一个work接收并执行xxxxxx,

获取执行结果

from celery.result import AsyncResult
from s1 import app

async = AsyncResult(id="f0b41e83-99cf-469f-9eff-74c8dd600002", app=app)
# AsyncResult(id="上面的resutl.id", app=Celery实例)

if async.successful(): # 判断是否执行完毕
    result = async.get() # async.get()是阻塞的,可以设置超时时间
    print(result)
    # result.forget() # 将结果删除
elif async.failed():
    print('执行失败')
elif async.status == 'PENDING':
    print('任务等待中被执行')
elif async.status == 'RETRY':
    print('任务异常后正在重试')
elif async.status == 'STARTED':
    print('任务已经开始被执行')

多实例应用同一方法

from celery import Celery
from celery import shared_task
app1 = Celery('tasks1', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')
app2 = Celery('tasks2', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')


@shared_task
def xxxxxx(x, y):
    return x + y

　　如果两个app的某个任务执行的是相同的操作,可以使用上面的方式. 该文件下的所有Celery对象都可调用xxxxxx方法,如果在该文件下为每个Celery实例注册(不同逻辑)任务,因为在同一名称空间内,会出现一些问题

# work
from celery import Celery
from celery import shared_task
app1 = Celery('tasks1', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')
app2 = Celery('tasks2', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')

@app1.task
def xxxxxx(x, y):
    return x + y

@app2.task
def xxxxxx(x, y):
    return x - y

# 发布任务
from s1 import xxxxxx
import datetime

# 构建一个utc时间对象
ctime = datetime.datetime.now()
utc_ctime = datetime.datetime.utcfromtimestamp(ctime.timestamp())
ctime_x = utc_ctime + datetime.timedelta(seconds=1)

result = xxxxxx.apply_async(args=[1, 3], eta=ctime_x)
print(type(result))
print(result.id)

# 获取到的结果
app1 : 执行完成并获取结果： -2
app2 : 执行完成并获取结果： -2

多任务结构

pro_cel
    ├── celery_tasks# celery相关文件夹
    │   ├── celery.py   # celery连接和配置相关文件,必须叫这个名
    │   └── tasks.py    #  所有任务函数
    ├── check_result.py # 检查结果
    └── send_task.py    # 触发任务

pro_cel/celery_tasks/celery

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from celery import Celery

celery = Celery('xxxxxx',
                broker='redis://192.168.0.111:6379',
                backend='redis://192.168.0.111:6379',
                include=['celery_tasks.tasks']) # 函数位置

# 时区
celery.conf.timezone = 'Asia/Shanghai'
# 是否使用UTC
celery.conf.enable_utc = False

pro_cel/celery_tasks/tasks.py

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import time
from .celery import celery


@celery.task
def xxxxx(*args, **kwargs):
    time.sleep(5)
    return "任务结果"


@celery.task
def hhhhhh(*args, **kwargs):
    time.sleep(5)
    return "任务结果"

pro_cel/check_result.py

#!/usr/bin/env python
# -*- coding:utf-8 -*-

from celery.result import AsyncResult
from celery_tasks.celery import celery

async = AsyncResult(id="ed88fa52-11ea-4873-b883-b6e0f00f3ef3", app=celery)

if async.successful():
    result = async.get()
    print(result)
    # result.forget() # 将结果删除
elif async.failed():
    print('执行失败')
elif async.status == 'PENDING':
    print('任务等待中被执行')
elif async.status == 'RETRY':
    print('任务异常后正在重试')
elif async.status == 'STARTED':
    print('任务已经开始被执行')

pro_cel/check_result.py

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import celery_tasks.tasks

# 立即告知celery去执行xxxxxx任务，并传入两个参数
result = celery_tasks.tasks.xxxxx.delay(4, 4)

print(result.id)

启动work

celery worker -A celery_tasks -l info

定时任务

设定时间让celery执行一个任务(需是utc时间)

from s1 import xxxxxx
# 告知celery，什么时候帮我执行任务
import datetime

# ctime = datetime.datetime(year=2019,month=2,day=8)
# 构建一个utc时间对象
ctime = datetime.datetime.now()
utc_ctime = datetime.datetime.utcfromtimestamp(ctime.timestamp())
ctime_x = utc_ctime + datetime.timedelta(seconds=1)

result = xxxxxx.apply_async(args=[1, 3], eta=ctime_x)
print(type(result))
print(result.id)

类似于contab的定时任务

from celery import Celery
from celery.schedules import crontab
 
app = Celery('tasks', broker='amqp://47.98.134.86:5672', backend='amqp://47.98.134.86:5672', include=['proj.s1', ])
app.conf.timezone = 'Asia/Shanghai'
app.conf.enable_utc = False
 
app.conf.beat_schedule = {
    # 'add-every-10-seconds': {
    #     'task': 'proj.s1.add1', # 执行方法
    #     'schedule': 10.0, # 时间设置/10秒
    #     'args': (16, 16) # 方法参数
    # },
    'add-every-12-seconds': {
        'task': 'proj.s1.add1',
        'schedule': crontab(minute=42, hour=8, day_of_month=11, month_of_year=4), # 哪月哪天哪时哪分执行
        'args': (16, 16)
    },
}