deque

deque

https://pymotw.com/2/collections/deque.html

A double-ended queue, or deque, supports adding and removing elements from either end. The more commonly used stacks and queues are degenerate forms of deques, where the inputs and outputs are restricted to a single end.

A deque can be populated from either end, termed “left” and “right” in the Python implementation.

import collections

# Add to the right
d = collections.deque()
d.extend('abcdefg')
print 'extend    :', d
d.append('h')
print 'append    :', d

# Add to the left
d = collections.deque()
d.extendleft('abcdefg')
print 'extendleft:', d
d.appendleft('h')
print 'appendleft:', d

deque是线程安全的吗？

由于GIL特性，有些接口是安全的。

但是其不是用于线程间交换数据的。

https://www.zhihu.com/question/52605823

作者：tomsheep
链接：https://www.zhihu.com/question/52605823/answer/155754207
来源：知乎
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。

一、首先第一个问题， deque源码里没有用锁保护critical section，是如何实现线程安全的？

看上去没有锁，但实际上是有的，没错就是GIL。

我们看Python文档中的解释

the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (see sys.setcheckinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

这段话的主要信息有：

1. 只有获取了GIL锁的线程才能操作Python对象或调用Python/C API

2. 什么时候会当前线程会释放GIL呢？一个是python解释器每执行若干条python指令（缺省值100）后会尝试做线程切换，让其他线程有机会执行。另一个是进行IO操作时，会释放掉GIL，当前线程进入睡眠。

那么deque的C语言实现中，以pop为例（deque_append_internal这个函数内容就不贴了）

static PyObject *
deque_append(dequeobject *deque, PyObject *item)
{
    Py_INCREF(item);
    if (deque_append_internal(deque, item, deque->maxlen) < 0)
        return NULL;
    Py_RETURN_NONE;
}

在通过Python/C API调用这个函数时，当前线程肯定是先拿到了GIL的，而在这个函数内部，没有主动释放GIL(如何释放，参考Initialization, Finalization, and Threads) 所以整个函数可以看作被GIL保护的critical section。

这里正好也有一个相关的讨论，Issue 15329: clarify which deque methods are thread-safe 可以拉到最后看Raymond的回答。

第二个问题，既然有GIL，那为啥在很多时候我们写的代码还要加锁？

因为——再重复一次上面提到的——python解释器每执行若干条python指令（缺省值100）后会尝试做线程切换（释放GIL）。注意跟上面用Python/C API实现的函数不同，C函数中如果没有主动释放GIL，是不会有机会切走的。

结论：

The deque's append(), appendleft(), pop(), popleft(), and len(d) operations are thread-safe in CPython

deque只能作为一种普通的队列数据结构

https://stackoverflow.com/questions/717148/queue-queue-vs-collections-deque/20330499#20330499

没有线程间通讯功能。

也不要将其在不同线程间共享，否则如同玩火。

 

Queue.Queue and collections.deque serve different purposes. Queue.Queue is intended for allowing different threads to communicate using queued messages/data, whereas collections.deque is simply intended as a datastructure. That's why Queue.Queue has methods like put_nowait(), get_nowait(), and join(), whereas collections.deque doesn't. Queue.Queue isn't intended to be used as a collection, which is why it lacks the likes of the in operator.

It boils down to this: if you have multiple threads and you want them to be able to communicate without the need for locks, you're looking for Queue.Queue; if you just want a queue or a double-ended queue as a datastructure, use collections.deque.

Finally, accessing and manipulating the internal deque of a Queue.Queue is playing with fire - you really don't want to be doing that.