前言
因为Redis的数据都储存在内存中，当进程退出时，所有数据都将丢失。为了保证数据安全，Redis支持RDB和AOF两种持久化机制有效避免数据丢失问题。RDB可以看作在某一时刻Redis的快照（snapshot），非常适合灾难恢复。AOF则是写入操作的日志。本文主要讲解RDB、AOF和混合结合使用。
一.探索RDB
RDB就像是一台给Redis内存数据存储拍照的照相机，生成快照保存到磁盘的过程。触发RDB持久化分为手动触发和自动触发。Redis重启读取RDB速度快，但是无法做到实时持久化，因此一般用于数据冷备和复制传输。
手动触发
使用save命令：此命令会使用Redis的主线程进程同步存储，阻塞当前的Redis服务器，造成服务不可用，直到RDB过程完成。无论当前服务器数据量大小，线上不要用。
127.0.0.1:6379> save OK
(1.14s) 59117:M 13 Apr 13:34:51.948 * DB saved on disk
使用bgsave命令：此命令会通过fork()创建子进程，在后台进程存储。只有fork阶段会阻塞当前Redis服务器，不必到整个RDB过程结束，一般时间很短。因此Redis内部涉及到RDB都采用bgsave命令。这里注意一点，无论RDB还是AOF，由于使用了写时复制，fork出来的子进程不需要拷贝父进程的物理内存空间，但是会复制父进程的空间内存页表。
127.0.0.1:6379> bgsave Background saving started 59117:M 13 Apr 13:44:40.312 * Background saving started by pid 59180 59180:C 13 Apr 13:44:40.314 * DB saved on disk 59117:M 13 Apr 13:44:40.317 * Background saving terminated with success
自动触发
一般我们是不会直接用命令生成RDB文件的，Redis支持自动触发RDB持久化机制，配置都在redis.conf文件里面，我们先来看一下文件里关于rdb的默认配置，这边都用红色字体标注出来了，英文的文档解释的十分清楚，注释也写的很不错。
################################ SNAPSHOTTING ################################ # # Save the DB on disk: # # save <seconds> <changes> # # Will save the DB if both the given number of seconds and the given # number of write operations against the DB occurred. # # In the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # Note: you can disable saving completely by commenting out all "save" lines. # # It is also possible to remove all the previously configured save # points by adding a save directive with a single empty string argument # like in the following example: # # save "" save 900 1 save 300 10 save 60 10000 # By default Redis will stop accepting writes if RDB snapshots are enabled # (at least one save point) and the latest background save failed. # This will make the user aware (in a hard way) that data is not persisting # on disk properly, otherwise chances are that no one will notice and some # disaster will happen. # # If the background saving process will start working again Redis will # automatically allow writes again. # # However if you have setup your proper monitoring of the Redis server # and persistence, you may want to disable this feature so that Redis will # continue to work as usual even if there are problems with disk, # permissions, and so forth. stop-writes-on-bgsave-error yes # Compress string objects using LZF when dump .rdb databases? # For default that's set to 'yes' as it's almost always a win. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. rdbcompression yes # Since version 5 of RDB a CRC64 checksum is placed at the end of the file. # This makes the format more resistant to corruption but there is a performance # hit to pay (around 10%) when saving and loading RDB files, so you can disable it # for maximum performances. # # RDB files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the check. rdbchecksum yes # The filename where to dump the DB dbfilename dump.rdb # The working directory. # # The DB will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # # The Append Only File will also be created inside this directory. # # Note that you must specify a directory here, not a file name. dir /usr/local/var/db/redis/
- save m n：代表Redis服务器在m秒内数据存在n次修改时，自动触发rdb。这个参数比较关键。
- stop-writes-on-bgsave-error：如果是yes，当bgsave命令失败时Redis将停止写入操作。
- rdbcompression：是否对RDB文件进行压缩，但是在LZF压缩消耗更多CPU
- rdbchecksum：是否对RDB文件进程校验
- dbfilename：配置文件名称，默认dump.rdb
- dir：配置rdb文件存放的路劲，这个参数比较重要。
工作原理
首先我们来看一下server.h文件内saveparams参数，可以看到，seconds就是秒数，changes就是改变量。是不是就对应着刚刚说的save m n的配置呢？
struct redisServer { .... struct saveparam *saveparams; /* Save points array for RDB */ ... }; struct saveparam { time_t seconds; int changes; };
接下来我们看这个redis.c文件，有个周期性函数，叫做serverCron，它会周期调用，大概做这几件事情，见注释。用红色标注的说明会触发bgsave和aof rewrite。
/* This is our timer interrupt, called server.hz times per second. * Here is where we do a number of things that need to be done asynchronously. * For instance: * * - Active expired keys collection (it is also performed in a lazy way on * lookup). * - Software watchdog. * - Update some statistic. * - Incremental rehashing of the DBs hash tables. * - Triggering BGSAVE / AOF rewrite, and handling of terminated children. * - Clients timeout of different kinds. * - Replication reconnection. * - Many more... * * Everything directly called here will be called server.hz times per second, * so in order to throttle execution of things we want to do less frequently * a macro is used: run_with_period(milliseconds) { .... } */ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
在这个方法里面有这样一段代码，这边单独拿出来，这段代码的意思是判断changes是否满足并执行save操作。
/* If there is not a background saving/rewrite in progress check if * we have to save/rewrite now */ for (j = 0; j < server.saveparamslen; j++) { struct saveparam *sp = server.saveparams+j; /* Save if we reached the given amount of changes, * the given amount of seconds, and if the latest bgsave was * successful or if, in case of an error, at least * CONFIG_BGSAVE_RETRY_DELAY seconds already elapsed. */ if (server.dirty >= sp->changes && server.unixtime-server.lastsave > sp->seconds && (server.unixtime-server.lastbgsave_try > CONFIG_BGSAVE_RETRY_DELAY || server.lastbgsave_status == C_OK)) { serverLog(LL_NOTICE,"%d changes in %d seconds. Saving...", sp->changes, (int)sp->seconds); rdbSaveBackground(server.rdb_filename); break; } }
接着继续看这个方法的部分代码片段，在rdb.c文件里。我们可以看到子进程名为"redis-rdb-bgsave"
int rdbSaveBackground(char *filename) { pid_t childpid; long long start; if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR; server.dirty_before_bgsave = server.dirty; server.lastbgsave_try = time(NULL); start = ustime(); if ((childpid = fork()) == 0) { int retval; /* Child */ closeListeningSockets(0); redisSetProcTitle("redis-rdb-bgsave"); retval = rdbSave(filename); if (retval == C_OK) { size_t private_dirty = zmalloc_get_private_dirty(); if (private_dirty) { serverLog(LL_NOTICE, "RDB: %zu MB of memory used by copy-on-write", private_dirty/(1024*1024)); } } exitFromChild((retval == C_OK) ? 0 : 1); }
最后我们看一下RDB的运作流程图：
- redis执行bgsave命令，Redis判断当前存在正在进行执行的子进程，如RDB/AOF子进程，存在bgsave命令直接返回
- fork出子进程，fork操作中Redis父进程会阻塞
- fork完成返回 59117:M 13 Apr 13:44:40.312 * Background saving started by pid 59180
- 子进程进程对内存数据生成快找文件
- 子进程告诉父进程处理完成
探索RDB文件
我们可以使用redis-rdb-tools来分析rdb快照文件，他可以把rdb快照文件生成json文件，看起来比较方便。
rdb -c memory dump.rdb > testMjx.csv
然后我们看下生成的文件长啥样
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry 0,string,mjx3,56,string,4,4, 0,string,mjx5,56,string,4,4, 0,string,mjx2,56,string,4,4, 0,string,mjx,48,string,8,8, 0,string,mjx4,56,string,4,4,
生成的数据有database（key在Redis的db）、type（key类型）、key（key值）、size_in_bytes（key的内存大小）、encoding（value的存储编码形式）、num_elements（key中的value的个数）、len_largest_element（key中的value的长度）、超时时间。
优缺点
RDB持久化方式的优点：
- 非常适合全量备份
- 恢复速度比AOF快
RDB持久化方式的缺点：
- RDB方式没有办法做到实时持久化
- 版本兼容RDB格式问题
二.探索AOF
RDB方式不能提供强一致性，如果Redis进程崩溃，那么两次RDB之间的数据也随之消失。那么AOF的出现很好的解决了数据持久化的实时性，AOF以独立日志的方式记录每次写命令，重启时再重新执行AOF文件中的命令来恢复数据。AOF会先把命令追加在AOF缓冲区，然后根据对应策略写入硬盘（appendfsync），具体参数后面有讲。接下来介绍一下AOF重写命令。
手动触发
使用bgrewriteaof命令：Redis主进程fork子进程来执行AOF重写，这个子进程创建新的AOF文件来存储重写结果，防止影响旧文件。因为fork采用了写时复制机制，子进程不能访问在其被创建出来之后产生的新数据。Redis使用“AOF重写缓冲区”保存这部分新数据，最后父进程将AOF重写缓冲区的数据写入新的AOF文件中然后使用新AOF文件替换老文件。
127.0.0.1:6379> bgrewriteoaf OK
自动触发
和RDB一样，配置在redis.conf文件里，当然你也可以通过调用CONFIG SET命令设置。我们先看来看AOF相关配置：
############################## APPEND ONLY MODE ############################### # By default Redis asynchronously dumps the dataset on disk. This mode is # good enough in many applications, but an issue with the Redis process or # a power outage may result into a few minutes of writes lost (depending on # the configured save points). # # The Append Only File is an alternative persistence mode that provides # much better durability. For instance using the default data fsync policy # (see later in the config file) Redis can lose just one second of writes in a # dramatic event like a server power outage, or a single write if something # wrong with the Redis process itself happens, but the operating system is # still running correctly. # # AOF and RDB persistence can be enabled at the same time without problems. # If the AOF is enabled on startup Redis will load the AOF, that is the file # with the better durability guarantees. # # Please check http://redis.io/topics/persistence for more information. appendonly no # The name of the append only file (default: "appendonly.aof") appendfilename "appendonly.aof" # The fsync() call tells the Operating System to actually write data on disk # instead of waiting for more data in the output buffer. Some OS will really flush # data on disk, some other OS will just try to do it ASAP. # # Redis supports three different modes: # # no: don't fsync, just let the OS flush the data when it wants. Faster. # always: fsync after every write to the append only log. Slow, Safest. # everysec: fsync only one time every second. Compromise. # # The default is "everysec", as that's usually the right compromise between # speed and data safety. It's up to you to understand if you can relax this to # "no" that will let the operating system flush the output buffer when # it wants, for better performances (but if you can live with the idea of # some data loss consider the default persistence mode that's snapshotting), # or on the contrary, use "always" that's very slow but a bit safer than # everysec. # # More details please check the following article: # http://antirez.com/post/redis-persistence-demystified.html # # If unsure, use "everysec". # appendfsync always appendfsync everysec # appendfsync no # When the AOF fsync policy is set to always or everysec, and a background # saving process (a background save or AOF log background rewriting) is # performing a lot of I/O against the disk, in some Linux configurations # Redis may block too long on the fsync() call. Note that there is no fix for # this currently, as even performing fsync in a different thread will block # our synchronous write(2) call. # # In order to mitigate this problem it's possible to use the following option # that will prevent fsync() from being called in the main process while a # BGSAVE or BGREWRITEAOF is in progress. # # This means that while another child is saving, the durability of Redis is # the same as "appendfsync none". In practical terms, this means that it is # possible to lose up to 30 seconds of log in the worst scenario (with the # default Linux settings). # # If you have latency problems turn this to "yes". Otherwise leave it as # "no" that is the safest pick from the point of view of durability. no-appendfsync-on-rewrite no # Automatic rewrite of the append only file. # Redis is able to automatically rewrite the log file implicitly calling # BGREWRITEAOF when the AOF log size grows by the specified percentage. # # This is how it works: Redis remembers the size of the AOF file after the # latest rewrite (if no rewrite has happened since the restart, the size of # the AOF at startup is used). # # This base size is compared to the current size. If the current size is # bigger than the specified percentage, the rewrite is triggered. Also # you need to specify a minimal size for the AOF file to be rewritten, this # is useful to avoid rewriting the AOF file even if the percentage increase # is reached but it is still pretty small. # # Specify a percentage of zero in order to disable the automatic AOF # rewrite feature. auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # An AOF file may be found to be truncated at the end during the Redis # startup process, when the AOF data gets loaded back into memory. # This may happen when the system where Redis is running # crashes, especially when an ext4 filesystem is mounted without the # data=ordered option (however this can't happen when Redis itself # crashes or aborts but the operating system still works correctly). # # Redis can either exit with an error when this happens, or load as much # data as possible (the default now) and start if the AOF file is found # to be truncated at the end. The following option controls this behavior. # # If aof-load-truncated is set to yes, a truncated AOF file is loaded and # the Redis server starts emitting a log to inform the user of the event. # Otherwise if the option is set to no, the server aborts with an error # and refuses to start. When the option is set to no, the user requires # to fix the AOF file using the "redis-check-aof" utility before to restart # the server. # # Note that if the AOF file will be found to be corrupted in the middle # the server will still exit with an error. This option only applies when # Redis will try to read more data from the AOF file but not enough bytes # will be found. aof-load-truncated yes # When rewriting the AOF file, Redis is able to use an RDB preamble in the # AOF file for faster rewrites and recoveries. When this option is turned # on the rewritten AOF file is composed of two different stanzas: # # [RDB file][AOF tail] # # When loading Redis recognizes that the AOF file starts with the "REDIS" # string and loads the prefixed RDB file, and continues loading the AOF # tail. # # This is currently turned off by default in order to avoid the surprise # of a format change, but will at some point be used as the default. aof-use-rdb-preamble no
- appendonly：是否打开AOF持久化功能
- appendfilename：AOF文件名称
- appendfsync：同步频率
- auto-aof-rewrite-min-size：如果文件大小小于此值不会触发AOF，默认64MB
- auto-aof-rewrite-percentage：Redis记录最近的一次AOF操作的文件大小，如果当前AOF文件大小增长超过这个百分比则触发一次重写，默认100
这里介绍一下appendfsync参数的可配置值
- always：命令写入aof缓冲区后，每一次写入都需要同步，直到写入磁盘（阻塞，系统调用fsync）结束后返回。显然和Redis高性能背道而驰，不建议配置
- everysec：命令写入aof缓冲区后，在写入系统缓冲区直接返回（系统调用write），然后有专门线程每秒执行写入磁盘（阻塞，系统调用fsync）后返回
- no：命令写入aof缓冲区后，在写入系统缓冲区直接返回（系统调用write）。之后写入磁盘（阻塞，系统调用fsync）的操作由操作系统负责，通常最长30s
工作原理
这里看一段aof.c的代码，我们可以看到fork出名为"redis-aof-rewrite"的子进程
/* This is how rewriting of the append only file in background works: * * 1) The user calls BGREWRITEAOF * 2) Redis calls this function, that forks(): * 2a) the child rewrite the append only file in a temp file. * 2b) the parent accumulates differences in server.aof_rewrite_buf. * 3) When the child finished '2a' exists. * 4) The parent will trap the exit code, if it's OK, will append the * data accumulated into server.aof_rewrite_buf into the temp file, and * finally will rename(2) the temp file in the actual file name. * The the new file is reopened as the new append only file. Profit! */ int rewriteAppendOnlyFileBackground(void) { pid_t childpid; long long start; if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR; if (aofCreatePipes() != C_OK) return C_ERR; start = ustime(); if ((childpid = fork()) == 0) { char tmpfile[256]; /* Child */ closeListeningSockets(0); redisSetProcTitle("redis-aof-rewrite"); snprintf(tmpfile,256,"temp-rewriteaof-bg-%d.aof", (int) getpid()); if (rewriteAppendOnlyFile(tmpfile) == C_OK) { size_t private_dirty = zmalloc_get_private_dirty(); if (private_dirty) { serverLog(LL_NOTICE, "AOF rewrite: %zu MB of memory used by copy-on-write", private_dirty/(1024*1024)); } exitFromChild(0); } else { exitFromChild(1); } }
...
...
同样我们也看一下AOF的运作流程图：
- 所有的写入命令追加到aof缓冲区
- AOF缓冲区根据对应appendfsync配置向硬盘做同步操作
- 定期对AOF文件进行重写
- Redis重启时，可以加载AOF文件进行数据恢复
探索AOF文件
首先打开aof功能
127.0.0.1:6379> CONFIG SET appendonly yes OK 59117:M 13 Apr 19:24:53.940 * Background append only file rewriting started by pid 59895 59117:M 13 Apr 19:24:53.964 * AOF rewrite child asks to stop sending diffs. 59895:C 13 Apr 19:24:53.965 * Parent agreed to stop sending diffs. Finalizing AOF... 59895:C 13 Apr 19:24:53.965 * Concatenating 0.00 MB of AOF diff received from parent. 59895:C 13 Apr 19:24:53.966 * SYNC append only file rewrite performed 59117:M 13 Apr 19:24:53.996 * Background AOF rewrite terminated with success 59117:M 13 Apr 19:24:53.996 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 59117:M 13 Apr 19:24:53.997 * Background AOF rewrite finished successfully
然后我们放一些数据，并执行bgrewriteaof命令
127.0.0.1:6379> CONFIG SET appendonly yes OK 127.0.0.1:6379> set miao 24 OK 127.0.0.1:6379> set miao 177 OK 127.0.0.1:6379> lpush mlist 1 (integer) 1 127.0.0.1:6379> lpush mlist 2 (integer) 2 127.0.0.1:6379> lpush mlist 3 (integer) 3 127.0.0.1:6379> keys * 1) "miao" 2) "mlist"
接下来看一下aof文件：
*2 $6 SELECT $1 0 *3 $3 SET $4 miao $3 177 *2 $6 SELECT $1 0 *3 $5 lpush $5 mlist $1 1 *3 $5 lpush $5 mlist $1 2 *3 $5 lpush $5 mlist $1 3
这时候我们手动执行aof重写命令：
127.0.0.1:6379> bgrewriteaof Background append only file rewriting started 59117:M 13 Apr 19:29:31.017 * 10 changes in 300 seconds. Saving... 59117:M 13 Apr 19:29:31.017 * Background saving started by pid 59905 59905:C 13 Apr 19:29:31.020 * DB saved on disk 59117:M 13 Apr 19:29:31.120 * Background saving terminated with success 59117:M 13 Apr 19:29:49.409 * Background append only file rewriting started by pid 59906 59117:M 13 Apr 19:29:49.433 * AOF rewrite child asks to stop sending diffs. 59906:C 13 Apr 19:29:49.433 * Parent agreed to stop sending diffs. Finalizing AOF... 59906:C 13 Apr 19:29:49.434 * Concatenating 0.00 MB of AOF diff received from parent. 59906:C 13 Apr 19:29:49.434 * SYNC append only file rewrite performed 59117:M 13 Apr 19:29:49.533 * Background AOF rewrite terminated with success 59117:M 13 Apr 19:29:49.533 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 59117:M 13 Apr 19:29:49.534 * Background AOF rewrite finished successfully
然后再看一下文件：
*2 $6 SELECT $1 0 *3 $3 SET $4 miao $3 177 *5 $5 RPUSH $5 mlist $1 3 $1 2 $1 1
为什么AOF文件会变小？为了解决AOF文件会越来越大，Redis引入重写机制来缩小文件体积，体积变小因为：
- 多条写入命令可以合并成一条。比如上面的lpush命令了3次，最后合并成1条
- 重写后AOF文件只保留最终数据的写入命令
优缺点
AOF持久化方式的优点：
- 做到最多丢失1-2s内的数据（最多丢失2s数据，因为AOF追加阻塞）
AOF持久化方式的缺点：
- AOF文件比RDB文件大
- 可能导致追加阻塞
参考：
书籍参考和上文一样