背景
redis一直以来都是以单线程模式运行，这里的单线程指网络IO和命令的执行部分。今年发布了6.0版本，加上了多线程来处理网络IO（read，write）和命令的解析。
单线程模式优缺点
这个想必大家都知道，简单介绍一下。
优点：
- 纯内存操作，CPU不是其性能瓶颈，开多个进程也可以更容易的使用多个CPU
- 无需考虑多线程同步，对开发友好
- 执行命令天然原子性
- 使用IO多路复用来处理大量连接，省去了线程上下文切换的时间
缺点：
- 耗时的操作将引起阻塞
- 单实例不能充分利用多核CPU（read/write还是需要CPU的参与在内核态与用户态之间copy数据）
redis网络IO模型简介
redis采用IO多路复用来管理多个网络连接，代码编写采用Reactor模式。
主线程是一个事件循环。
简单看下源代码：
/* State of an event based program */
typedef struct aeEventLoop {
int maxfd; /* highest file descriptor currently registered */
int setsize; /* max number of file descriptors tracked */
long long timeEventNextId;
time_t lastTime; /* Used to detect system clock skew */
aeFileEvent *events; /* Registered events */
aeFiredEvent *fired; /* Fired events */
aeTimeEvent *timeEventHead;
int stop;
void *apidata; /* This is used for polling API specific data */
aeBeforeSleepProc *beforesleep;
aeBeforeSleepProc *aftersleep;
int flags;
} aeEventLoop;
struct redisServer {
aeEventLoop *el;
}
el变量保存了事件循环相关的信息，其中void *apidata;保存了IO多路复用API相关的信息，redis封装了select、epoll、kqueue等多种不同的IO多路复用函数，在编译期根据平台类型来选择一种。
ae.c:
/* Include the best multiplexing layer supported by this system.
* The following should be ordered by performances, descending. */
#ifdef HAVE_EVPORT
#include "ae_evport.c"
#else
#ifdef HAVE_EPOLL
#include "ae_epoll.c"
#else
#ifdef HAVE_KQUEUE
#include "ae_kqueue.c"
#else
#include "ae_select.c"
#endif
#endif
#endif
FileEvent
FileEvent其实就是网络IO事件，为fd绑定读写对应的事件处理函数，当通过IO多路复用获取到其就绪时，调用其绑定的处理函数。
/* File event structure */
typedef struct aeFileEvent {
int mask; /* one of AE_(READABLE|WRITABLE|BARRIER) */
aeFileProc *rfileProc;
aeFileProc *wfileProc;
void *clientData;
} aeFileEvent;
TimeEvent
TimeEvent是定时任务事件。每个定时任务都绑定一个执行函数，巧妙的利用IO多路复用API拉取就绪事件时的阻塞时间参数，来实现定时的效果。比如最近要执行的定时任务是100ms后（这里用的循环遍历的方式获取最值，时间复杂度O(n)，可以改用跳表之类的数据结构优化到O(log n)，应该是作者考虑到定时任务并不会特别多，所以这里并没有专门去做优化），那么就让select函数的阻塞超时时间设为100ms，这样就可以实现一个不是特别精确的定时器。
/* Time event structure */
typedef struct aeTimeEvent {
long long id; /* time event identifier. */
long when_sec; /* seconds */
long when_ms; /* milliseconds */
aeTimeProc *timeProc;
aeEventFinalizerProc *finalizerProc;
void *clientData;
struct aeTimeEvent *prev;
struct aeTimeEvent *next;
int refcount; /* refcount to prevent timer events from being
* freed in recursive time event calls. */
} aeTimeEvent;
6.0版本引入多线程
IO多线程相关的配置
先看一下6.0版本配置文件中关于多线程的参数和说明：
################################ THREADED I/O #################################
# Redis is mostly single threaded, however there are certain threaded
# operations such as UNLINK, slow I/O accesses and other things that are
# performed on side threads.
#
# Now it is also possible to handle Redis clients socket reads and writes
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speed up the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usual.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Aso this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis threads, otherwise you'll not
# be able to notice the improvements.
这里我们需要关注以下几点：
- 默认是单线程模式
- IO多线程用于
read、write函数 - 不需要开多个实例运行redis也可以轻松加速2倍的速度
io-threads参数指明有几个IO线程- 如果
io-threads是1，则只有一个主线程，如果是2，则多开一个IO线程，以此类推 - 默认只有
write函数会使用多线程 io-threads-do-reads控制read是否开启多线程- 多线程IO对read的帮助并不是特别大
- SSL模式暂时不支持这个配置
- 对多线程IO的redis做基准测试的时候，
redis-benchmark也要开启多线程参数
看看源码
IO线程的主要源代码在这里： networking.c#L2979
关键全局变量
pthread_t io_threads[IO_THREADS_MAX_NUM];
pthread_mutex_t io_threads_mutex[IO_THREADS_MAX_NUM];
_Atomic unsigned long io_threads_pending[IO_THREADS_MAX_NUM];
int io_threads_op; /* IO_THREADS_OP_WRITE or IO_THREADS_OP_READ. */
/* This is the list of clients each thread will serve when threaded I/O is
* used. We spawn io_threads_num-1 threads, since one is the main thread
* itself. */
list *io_threads_list[IO_THREADS_MAX_NUM];
io_threads：pthread多线程结构体
io_threads_mutex：互斥锁，用于在主线程控制IO线程的停止和运行
io_threads_pending：原子类型，和主线程进行同步的变量，如果io_threads_pending[i]==1说明编号为i的线程就绪了，可以进行读/写操作。
io_threads_op：当前操作是读还是写
io_threads_list：每个线程的client队列，对于某个thread，遍历list依次处理其下的client
关键代码逻辑
在beforeSleep函数中分别调用handleClientsWithPendingReadsUsingThreads和handleClientsWithPendingWritesUsingThreads来唤醒IO线程，分别处理读和写。
在handleClientsWithPendingReadsUsingThreads函数中可以看到，就绪的客户端会均匀分配到n个IO线程中去执行：
/* Distribute the clients across N different lists. */
listIter li;
listNode *ln;
listRewind(server.clients_pending_read,&li);
int item_id = 0;
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
int target_id = item_id % server.io_threads_num;
listAddNodeTail(io_threads_list[target_id],c);
item_id++;
}
然后会通过设置io_threads_pending变量来唤醒IO线程，假设设置了io-threads=4则会有io-threads - 1 = 3个额外的线程启动，因为主线程也会作为一个IO线程。主线程处理io_threads_list[0]里面的客户端。
/* Give the start condition to the waiting threads, by setting the
* start condition atomic var. */
io_threads_op = IO_THREADS_OP_READ;
for (int j = 1; j < server.io_threads_num; j++) {
int count = listLength(io_threads_list[j]);
io_threads_pending[j] = count;
}
/* Also use the main thread to process a slice of clients. */
listRewind(io_threads_list[0],&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
readQueryFromClient(c->conn);
}
listEmpty(io_threads_list[0]);
然后主线程做完IO操作之后，会死循环等待其他IO线程完成读操作，才会执行命令的执行，这个时候读取数据和解析命令已经在IO线程中完成了，主线程执行命令，保证了命令执行的原子性。
/* Wait for all the other threads to end their work. */
while(1) {
unsigned long pending = 0;
for (int j = 1; j < server.io_threads_num; j++)
pending += io_threads_pending[j];
if (pending == 0) break;
}
if (tio_debug) printf("I/O READ All threads finshed
");
/* Run the list of clients again to process the new buffers. */
while(listLength(server.clients_pending_read)) {
ln = listFirst(server.clients_pending_read);
client *c = listNodeValue(ln);
c->flags &= ~CLIENT_PENDING_READ;
listDelNode(server.clients_pending_read,ln);
if (c->flags & CLIENT_PENDING_COMMAND) {
c->flags &= ~CLIENT_PENDING_COMMAND;
if (processCommandAndResetClient(c) == C_ERR) {
/* If the client is no longer valid, we avoid
* processing the client later. So we just go
* to the next. */
continue;
}
}
processInputBuffer(c);
}
IO线程的执行逻辑在IOThreadMain中：
死循环中等待io_threads_pending被设置为非零值，这里如果死循环一直轮询会把CPU吃满，所以这里还有一个互斥锁io_threads_mutex来暂停IO线程，使其阻塞在pthread_mutex_lock这里。
/* Wait for start */
for (int j = 0; j < 1000000; j++) {
if (io_threads_pending[id] != 0) break;
}
/* Give the main thread a chance to stop this thread. */
if (io_threads_pending[id] == 0) {
pthread_mutex_lock(&io_threads_mutex[id]);
pthread_mutex_unlock(&io_threads_mutex[id]);
continue;
}
接下来就是根据io_threads_op来区分是读还是写，去执行read或write
/* Process: note that the main thread will never touch our list
* before we drop the pending count to 0. */
listIter li;
listNode *ln;
listRewind(io_threads_list[id],&li);
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
if (io_threads_op == IO_THREADS_OP_WRITE) {
writeToClient(c,0);
} else if (io_threads_op == IO_THREADS_OP_READ) {
readQueryFromClient(c->conn);
} else {
serverPanic("io_threads_op value is unknown");
}
}
listEmpty(io_threads_list[id]);
io_threads_pending[id] = 0;
最后看下后台进程
io-threads=4，会多额外的3个io-thread：
top -Hp 17339
IO多线程模式流程
性能测试
这里使用redis自带的基准测试工具redis-benchmark来进行测试。
对比图
详情数据
redis6.0.9：IO线程数是4
多线程：./redis-benchmark -c 1000 -n 1000000 --threads 4 --csv
单线程：./redis-benchmark -c 1000 -n 1000000 --csv
表格：
| cmd | 4 threads & read yes | 4 threads & read no | 1 thread |
|---|---|---|---|
| PING_INLINE | 472589.81 | 363240.09 | 215610.17 |
| PING_BULK | 515198.34 | 423908.44 | 213766.56 |
| SET | 442673.75 | 372162.25 | 213401.62 |
| GET | 476644.41 | 400320.28 | 212901.84 |
| INCR | 460829.47 | 389559.81 | 214408.23 |
| LPUSH | 399520.56 | 346500.34 | 220896.84 |
| RPUSH | 430292.62 | 358680.03 | 217391.31 |
| LPOP | 404203.72 | 344946.53 | 222024.86 |
| RPOP | 399680.25 | 333111.25 | 215517.25 |
| SADD | 450856.66 | 363372.09 | 216590.86 |
| HSET | 399680.25 | 333111.25 | 217344.06 |
| SPOP | 486854.94 | 405350.62 | 213401.62 |
| ZADD | 415627.62 | 333222.28 | 217912.39 |
| ZPOPMIN | 444049.72 | 402900.88 | 216122.77 |
| LPUSH (needed to benchmark LRANGE) | 410677.62 | 342114.25 | 218914.19 |
| LRANGE_100 (first 100 elements) | 113869.28 | 110168.56 | 75483.09 |
| LRANGE_300 (first 300 elements) | 45687.13 | 44081.99 | 27139.99 |
| LRANGE_500 (first 450 elements) | 31991.81 | 31406.05 | 20085.56 |
| LRANGE_600 (first 600 elements) | 24688.31 | 23973.34 | 15635.75 |
| MSET (10 keys) | 226244.34 | 200240.30 | 175500.17 |
总结
redis6.0之后针对网络IO增加了多线程，IO线程中只负责read、解析command、write操作，命令执行操作还是在主线程，依然具有原子性。
开启四个IO线程的情况下，GET和SET操作，相对于单线程模式，开启write+read多线程，性能为原来的2倍，只开启write多线程，性能为原来的1.68倍
最后大家可以考虑下，为肾么，多线程执行read速度提升并不明显？