Storm-源码分析- Disruptor在storm中的使用

Disruptor 2.0, (http://ifeve.com/disruptor-2-change/)

Disruptor为了更便于使用, 在2.0做了比较大的调整, 比较突出的是更换了几乎所有的概念名

老版本,

新版本,

从左到右的变化如下,

1. Producer –> Publisher
2. ProducerBarrier被integrate到RingBuffer里面, 叫做PublishPort, 提供publish接口
3. Entry –> Event
4, Cursor封装成Sequence, 其实Sequence就是将cursor+pading封装一下
5. Consumer –> EventProcesser
6. ConsumerBarrier 变为DependencyBarrier, 或SequenceBarrier

并且对于publisher和EventProcesser, 存在ClaimStrategy和WaitStrategy
对于publisher的ClaimStrategy, 由于publisher需要先claim到sequencer才能publish: SingleThreadedClaimStrategy, MultiThreadedClaimStrategy, 应该是对于singlethread不需要使用CAS更为高效
对于EventProcesser的WaitStrategy, 当取不到数据的时候采用什么样的策略进行等待: BlockingWaitStrategy, BusySpinWaitStrategy, SleepingWaitStrategy, YieldingWaitStrategy
Blocking就是同步加锁, BusySpin就是忙等耗CPU, 都比较低效
Yielding就是调用thread.yeild(), 把线程的从可执行状态调整成就绪装, 意思我先息下, 你们忙你们先来, 就是把CPU让给其他的线程, 但是yeild并不保证过多久线程被执行, 如果没有其他线程, 可能会被立即执行
而sleep, 会强制线程休眠指定时间, 然后再重新调度

 

DisruptorQueue.java

    static final Object FLUSH_CACHE = new Object(); //特殊对象, 当consumer取到时, 触发cache queue的flush
    static final Object INTERRUPT = new Object(); //特殊对象, 当consumer取到时, 触发InterruptedException
    
    RingBuffer<MutableObject> _buffer; //Disruptor的主要的数据结构RingBuffer
    Sequence _consumer; //consumer读取序号
    SequenceBarrier _barrier; //用于consumer监听RingBuffer的序号情况
    
    // TODO: consider having a threadlocal cache of this variable to speed up reads?
    volatile boolean consumerStartedFlag = false; //标志consumer是否start, 由于需要在change后其他线程可以马上知道, 所以使用volatile
    ConcurrentLinkedQueue<Object> _cache = new ConcurrentLinkedQueue(); //当consumer没有start的时候, cache event的queue

ConcurrentLinkedQueue, 使用CAS而非lock来实现的线程安全队列, 具体参考(http://blog.sina.com.cn/s/blog_5efa3473010129pj.html)

首先声明一组变量, 部分会在构造函数中被初始化
最重要的结构就是RingBuffer, 这是个模板类, 这里从ObjectEventFactory()的实现也可以看出来, 初始化的时候在ringbuffer的每个entry上都创建一个MutableObject对象

MutableObject的实现很简单, 这是封装了object o, 为什么要做这层封装?
为了避免Java GC, 对于RingBuffer一旦初始化好, 上面的所有的MutableObject都不会被释放, 你只是去对object o, set不同的值

_buffer = new RingBuffer<MutableObject>(new ObjectEventFactory(), claim, wait);

public static class ObjectEventFactory implements EventFactory<MutableObject> {
    @Override
    public MutableObject newInstance() {
        return new MutableObject();
    }        
}

public class MutableObject {
    Object o = null;
}

Publish

Publish过程, 可见当前ProducerBarrier已经被集成到RingBuffer里面, 所以直接调用_buffer的接口
首先调用next, claim序号
取出序号上的MutableObject, 并将输入obj set
最后, publish当前序号, 表示consumer可以读取
当consumer没有start时, 会将obj cache在_cache中, 而不会放到ringbuffer中 (我没有想明白why? 为何要使用低效的链表queue来cache, 而不直接放到ringbuffer里面)

    public void publish(Object obj, boolean block) throws InsufficientCapacityException {
        if(consumerStartedFlag) {
            final long id;
            if(block) {
                id = _buffer.next();
            } else {
                id = _buffer.tryNext(1);
            }
            final MutableObject m = _buffer.get(id);
            m.setObject(obj);
            _buffer.publish(id);
        } else {
            _cache.add(obj);
            if(consumerStartedFlag) flushCache();
        }
    }

Consume

consume的过程, 这里实现的时Batch consume, 即给定Cursor, 会一直consume到该cursor为止

_consumer代表当前已经被consume的序号, 所以从_consumer.get() + 1开始读
取出MutableObject中的o, 并将MutableObject 清空
根据o的情况, 3种情况,
    1. 如果是FLUSH_CACHE对象, 将cache中的event读出调用handler.onEvent
    2. 如果是INTERRUPT对象, 触发InterruptedException
    3. 正常情况, 直接调用handler.onEvent处理该o, curr == cursor判断表示batch是否结束, 当读到cursor的时候结束

最终将_consumer置为cursor, 表示已经读到cursor位置

    private void consumeBatchToCursor(long cursor, EventHandler<Object> handler) {
        for(long curr = _consumer.get() + 1; curr <= cursor; curr++) {
            try {
                MutableObject mo = _buffer.get(curr);
                Object o = mo.o;
                mo.setObject(null);
                if(o==FLUSH_CACHE) {
                    Object c = null;
                    while(true) {                        
                        c = _cache.poll();
                        if(c==null) break;
                        else handler.onEvent(c, curr, true);
                    }
                } else if(o==INTERRUPT) {
                    throw new InterruptedException("Disruptor processing interrupted");
                } else {
                    handler.onEvent(o, curr, curr == cursor);
                }
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
        //TODO: only set this if the consumer cursor has changed?
        _consumer.set(cursor);
    }

backtype.storm.disruptor.clj

创建DisruptorQueue, 选用MultiThreadedClaimStrategy和BlockingWaitStrategy

(defnk disruptor-queue [buffer-size :claim-strategy :multi-threaded :wait-strategy :block]
  (DisruptorQueue. ((CLAIM-STRATEGY claim-strategy) buffer-size)
                   (mk-wait-strategy wait-strategy)
                   ))

并封装一系列Java接口
最重要的工作是, 启动consume-loop
这里ret是closeover了一个间隔为0的不停执行(consume-batch-when-available queue handler) 的线程, 而consumeBatchWhenAvailable的实现就是不停的sleep并调用consumeBatchToCursor

并且通过consumer-started!通知其他线程consumer已经start

(defnk consume-loop* [^DisruptorQueue queue handler :kill-fn (fn [error] (halt-process! 1 "Async loop died!"))
                      :thread-name nil]
  (let [ret (async-loop
              (fn []
                (consume-batch-when-available queue handler)
                0 )
              :kill-fn kill-fn
              :thread-name thread-name
              )]
     (consumer-started! queue)
     ret
     ))

(defmacro consume-loop [queue & handler-args]
  `(let [handler# (handler ~@handler-args)]
     (consume-loop* ~queue handler#)
     ))

看看async-loop实现什么功能?
返回reify实现的record, 其中closeover了thread
这个thread主要就是死循环的执行传入的afn, 并且以afn的返回值作为执行间隔

主要功能, 异步的loop, 开启新的线程来执行loop, 而不是在当前主线程, 并且提供了sleep设置

;; afn returns amount of time to sleep
(defnk async-loop [afn
                   :daemon false
                   :kill-fn (fn [error] (halt-process! 1 "Async loop died!"))
                   :priority Thread/NORM_PRIORITY
                   :factory? false
                   :start true
                   :thread-name nil]
  (let [thread (Thread.
                (fn []
                  (try-cause
                    (let [afn (if factory? (afn) afn)]
                      (loop []
                        (let [sleep-time (afn)]
                          (when-not (nil? sleep-time)
                            (sleep-secs sleep-time)
                            (recur))
                          )))
                    (catch InterruptedException e
                      (log-message "Async loop interrupted!")
                      )
                    (catch Throwable t
                      (log-error t "Async loop died!")
                      (kill-fn t)
                      ))
                  ))]
    (.setDaemon thread daemon)
    (.setPriority thread priority)
    (when thread-name
      (.setName thread (str (.getName thread) "-" thread-name)))
    (when start
      (.start thread))
    ;; should return object that supports stop, interrupt, join, and waiting?
    (reify SmartThread
      (start [this]
        (.start thread))
      (join [this]
        (.join thread))
      (interrupt [this]
        (.interrupt thread))
      (sleeping? [this]
        (Time/isThreadWaiting thread)
        ))
      ))

 

Storm在Worker中executors线程间通信, 如何使用Disruptor的?

 

Understanding the Internal Message Buffers of Storm, 可以参考