zookeeper(3) 持久化

zookeeper为了防止，系统宕机或重启导致的数据丢失，会对数据进行定时持久化。有两种持久化方式：

　　1.为每次事务操作记录到日志文件，这样就可以通过执行这些日志文件来恢复数据。

　　2.为了加快ZooKeeper恢复的速度，ZooKeeper还提供了对树结构和session信息进行数据快照持久化的操作。

日志文件

 　  日志文件记录zookeeper服务器上的每一次事务操作。

　　日志文件格式：log.ZXID，ZXID非常重要，它表示该文件起始的事务id。

数据快照

　　数据快照用来记录zookeeper服务器上某一时刻的全量内存数据内容，并写入指定磁盘文件中。

　　数据快照文件格式：snapshot.ZXID，ZXID非常重要，ZooKeeper会根据ZXID来确定数据恢复的起始点。

　　镜像文件主要存储zookeeper的树结构和session信息。

类图

FileTxnSnapLog

　　是操作数据持久化的核心类，底层通过TxnLog和SnapShot来分别操作日志文件和数据快照。

　　存储数据快照

public void save(DataTree dataTree,
            ConcurrentHashMap<Long, Integer> sessionsWithTimeouts)
        throws IOException {
        long lastZxid = dataTree.lastProcessedZxid;
        LOG.info("Snapshotting: " + Long.toHexString(lastZxid));
        File snapshot=new File(
                snapDir, Util.makeSnapshotName(lastZxid));
        snapLog.serialize(dataTree, sessionsWithTimeouts, snapshot);
        
    }

 　　日志文件操作

public boolean append(Request si) throws IOException {
        return txnLog.append(si.hdr, si.txn);
    }
    public void commit() throws IOException {
        txnLog.commit();
    }
    public void rollLog() throws IOException {
        txnLog.rollLog();
    }

　　数据恢复

public long restore(DataTree dt, Map<Long, Integer> sessions, 
            PlayBackListener listener) throws IOException {
        snapLog.deserialize(dt, sessions);
        FileTxnLog txnLog = new FileTxnLog(dataDir);
        TxnIterator itr = txnLog.read(dt.lastProcessedZxid+1);
        long highestZxid = dt.lastProcessedZxid;
        TxnHeader hdr;
        while (true) {
            
            hdr = itr.getHeader();
            ...if (hdr.getZxid() < highestZxid && highestZxid != 0) {
                LOG.error(highestZxid + "(higestZxid) > "
                        + hdr.getZxid() + "(next log) for type "
                        + hdr.getType());
            } else {
                highestZxid = hdr.getZxid();
            }
            try {
                processTransaction(hdr,dt,sessions, itr.getTxn());
            } catch(KeeperException.NoNodeException e) {
                throw new IOException("Failed to process transaction type: " +
                        hdr.getType() + " error: " + e.getMessage());
            }
            if (!itr.next()) 
                break;
        }
        return highestZxid;
    }
    

 FileTxnLog

　　负责维护事务日志对外的接口，包括事务日志的写入和读取等。

　　写入事务日志

　　1.如果日志文件打开，使用该日志文件；如果没有，使用该事务的zxid做为后缀，创建新的日志文件。

　　2.如果当前日志文件剩余空间不足4kb，对日志文件扩容到64mb，使用0来填充。预分配的原因是提高io效率。

　　3.对事务的头和事务体序列号

　　4.生成checksum

　　5.写入文件流。

public synchronized boolean append(TxnHeader hdr, Record txn)
        throws IOException
    {
        if (hdr != null) {
            ...
            if (logStream==null) {
                ...
               logFileWrite = new File(logDir, ("log." + 
                       Long.toHexString(hdr.getZxid())));
               fos = new FileOutputStream(logFileWrite);
               logStream=new BufferedOutputStream(fos);
               oa = BinaryOutputArchive.getArchive(logStream);
               FileHeader fhdr = new FileHeader(TXNLOG_MAGIC,VERSION, dbId);
               fhdr.serialize(oa, "fileheader");
               // Make sure that the magic number is written before padding.
               logStream.flush();
               currentSize = fos.getChannel().position();
               streamsToFlush.add(fos);
            }
            padFile(fos);
            byte[] buf = Util.marshallTxnEntry(hdr, txn);
            ...
            Checksum crc = makeChecksumAlgorithm();
            crc.update(buf, 0, buf.length);
            oa.writeLong(crc.getValue(), "txnEntryCRC");
            Util.writeTxnBytes(oa, buf);
            return true;
        }
        return false;
    }

　　持久化本质是将内存中对象数据以二进制的方式存储到磁盘上，这个过程，底层通过jute来序列号。

　　序列化和反序列化的本质就是数据流与对象数据之间的变换。jute的序列化理念是让需要序列化的对象自己定义序列化协议。所以使用jute进行序列化的对象需要实现Record接口,该接口需要对象实现序列化和反序列化方法。此外jute还对序列化的流进行了抽象，OutputArchive代表输入流，InputArchive代表输入流，各种类型流的读写通过实现这两个接口实现。通过实现Record接口，对象定义序列化和反序列化的协议；通过实现OutputArchive和InputArchive，实现数据存储和读取。

Record代码：

public interface Record {
    public void serialize(OutputArchive archive, String tag)
        throws IOException;
    public void deserialize(InputArchive archive, String tag)
        throws IOException;
}

OutputArchive代码：

public interface OutputArchive {
    public void writeByte(byte b, String tag) throws IOException;
    public void writeBool(boolean b, String tag) throws IOException;
    public void writeInt(int i, String tag) throws IOException;
    public void writeLong(long l, String tag) throws IOException;
    public void writeFloat(float f, String tag) throws IOException;
    public void writeDouble(double d, String tag) throws IOException;
    public void writeString(String s, String tag) throws IOException;
    public void writeBuffer(byte buf[], String tag)
        throws IOException;
    public void writeRecord(Record r, String tag) throws IOException;
    public void startRecord(Record r, String tag) throws IOException;
    public void endRecord(Record r, String tag) throws IOException;
    public void startVector(List v, String tag) throws IOException;
    public void endVector(List v, String tag) throws IOException;
    public void startMap(TreeMap v, String tag) throws IOException;
    public void endMap(TreeMap v, String tag) throws IOException;

}

InputArchive代码：

public interface InputArchive {
    public byte readByte(String tag) throws IOException;
    public boolean readBool(String tag) throws IOException;
    public int readInt(String tag) throws IOException;
    public long readLong(String tag) throws IOException;
    public float readFloat(String tag) throws IOException;
    public double readDouble(String tag) throws IOException;
    public String readString(String tag) throws IOException;
    public byte[] readBuffer(String tag) throws IOException;
    public void readRecord(Record r, String tag) throws IOException;
    public void startRecord(String tag) throws IOException;
    public void endRecord(String tag) throws IOException;
    public Index startVector(String tag) throws IOException;
    public void endVector(String tag) throws IOException;
    public Index startMap(String tag) throws IOException;
    public void endMap(String tag) throws IOException;
}

　　例如对FileHeader实现序列化，分别在serialize和deserialize方法中定义序列化协议，然后调用相应方法就可以将该对象序列化和反序列化。

public class FileHeader implements Record {
  private int magic;
  private int version;
  private long dbid;
  public void serialize(OutputArchive a_, String tag) throws java.io.IOException {
    a_.startRecord(this,tag);
    a_.writeInt(magic,"magic");
    a_.writeInt(version,"version");
    a_.writeLong(dbid,"dbid");
    a_.endRecord(this,tag);
  }
  public void deserialize(InputArchive a_, String tag) throws java.io.IOException {
    a_.startRecord(tag);
    magic=a_.readInt("magic");
    version=a_.readInt("version");
    dbid=a_.readLong("dbid");
    a_.endRecord(tag);
}
}

　　具体对象会序列化为什么样的数据形式以及从什么样数据形式中反序列化，取决于OutputArchive和InputArchive的实现。

二进制数据流实现：

BinaryOutputArchive：

public class BinaryOutputArchive implements OutputArchive {
    private ByteBuffer bb = ByteBuffer.allocate(1024);
    private DataOutput out;
    public static BinaryOutputArchive getArchive(OutputStream strm) {
        return new BinaryOutputArchive(new DataOutputStream(strm));
    }
    public BinaryOutputArchive(DataOutput out) {
        this.out = out;
    }
    public void writeByte(byte b, String tag) throws IOException {
        out.writeByte(b);
    }
    public void writeBool(boolean b, String tag) throws IOException {
        out.writeBoolean(b);
    }
    public void writeInt(int i, String tag) throws IOException {
        out.writeInt(i);
    }
    public void writeLong(long l, String tag) throws IOException {
        out.writeLong(l);
    }
    public void writeFloat(float f, String tag) throws IOException {
        out.writeFloat(f);
    }
    public void writeDouble(double d, String tag) throws IOException {
        out.writeDouble(d);
    }
    
    /**
     * create our own char encoder to utf8. This is faster 
     * then string.getbytes(UTF8).
     * @param s the string to encode into utf8
     * @return utf8 byte sequence.
     */
    final private ByteBuffer stringToByteBuffer(CharSequence s) {
        bb.clear();
        final int len = s.length();
        for (int i = 0; i < len; i++) {
            if (bb.remaining() < 3) {
                ByteBuffer n = ByteBuffer.allocate(bb.capacity() << 1);
                bb.flip();
                n.put(bb);
                bb = n;
            }
            char c = s.charAt(i);
            if (c < 0x80) {
                bb.put((byte) c);
            } else if (c < 0x800) {
                bb.put((byte) (0xc0 | (c >> 6)));
                bb.put((byte) (0x80 | (c & 0x3f)));
            } else {
                bb.put((byte) (0xe0 | (c >> 12)));
                bb.put((byte) (0x80 | ((c >> 6) & 0x3f)));
                bb.put((byte) (0x80 | (c & 0x3f)));
            }
        }
        bb.flip();
        return bb;
    }

    public void writeString(String s, String tag) throws IOException {
        if (s == null) {
            writeInt(-1, "len");
            return;
        }
        ByteBuffer bb = stringToByteBuffer(s);
        writeInt(bb.remaining(), "len");
        out.write(bb.array(), bb.position(), bb.limit());
    }

    public void writeBuffer(byte barr[], String tag)
    throws IOException {
        if (barr == null) {
            out.writeInt(-1);
            return;
        }
        out.writeInt(barr.length);
        out.write(barr);
    }
    
    public void writeRecord(Record r, String tag) throws IOException {
        r.serialize(this, tag);
    }
    public void startRecord(Record r, String tag) throws IOException {}
    
    public void endRecord(Record r, String tag) throws IOException {}
    
    public void startVector(List v, String tag) throws IOException {
        if (v == null) {
            writeInt(-1, tag);
            return;
        }
        writeInt(v.size(), tag);
    }
    public void endVector(List v, String tag) throws IOException {}
    
    public void startMap(TreeMap v, String tag) throws IOException {
        writeInt(v.size(), tag);
    }
    public void endMap(TreeMap v, String tag) throws IOException {}
}

BinaryInputArchive：

public class BinaryInputArchive implements InputArchive {
    
    private DataInput in;
    
    static public BinaryInputArchive getArchive(InputStream strm) {
        return new BinaryInputArchive(new DataInputStream(strm));
    }
    
    static private class BinaryIndex implements Index {
        private int nelems;
        BinaryIndex(int nelems) {
            this.nelems = nelems;
        }
        public boolean done() {
            return (nelems <= 0);
        }
        public void incr() {
            nelems--;
        }
    }
    /** Creates a new instance of BinaryInputArchive */
    public BinaryInputArchive(DataInput in) {
        this.in = in;
    }
    
    public byte readByte(String tag) throws IOException {
        return in.readByte();
    }
    
    public boolean readBool(String tag) throws IOException {
        return in.readBoolean();
    }
    
    public int readInt(String tag) throws IOException {
        return in.readInt();
    }
    
    public long readLong(String tag) throws IOException {
        return in.readLong();
    }
    
    public float readFloat(String tag) throws IOException {
        return in.readFloat();
    }
    
    public double readDouble(String tag) throws IOException {
        return in.readDouble();
    }
    
    public String readString(String tag) throws IOException {
        int len = in.readInt();
        if (len == -1) return null;
        byte b[] = new byte[len];
        in.readFully(b);
        return new String(b, "UTF8");
    }
    
    static public final int maxBuffer = determineMaxBuffer();
    private static int determineMaxBuffer() {
        String maxBufferString = System.getProperty("jute.maxbuffer");
        try {
            return Integer.parseInt(maxBufferString);
        } catch(Exception e) {
            return 0xfffff;
        }
        
    }
    public byte[] readBuffer(String tag) throws IOException {
        int len = readInt(tag);
        if (len == -1) return null;
        if (len < 0 || len > maxBuffer) {
            throw new IOException("Unreasonable length = " + len);
        }
        byte[] arr = new byte[len];
        in.readFully(arr);
        return arr;
    }
    
    public void readRecord(Record r, String tag) throws IOException {
        r.deserialize(this, tag);
    }
    
    public void startRecord(String tag) throws IOException {}
    
    public void endRecord(String tag) throws IOException {}
    
    public Index startVector(String tag) throws IOException {
        int len = readInt(tag);
        if (len == -1) {
            return null;
        }
        return new BinaryIndex(len);
    }
    
    public void endVector(String tag) throws IOException {}
    
    public Index startMap(String tag) throws IOException {
        return new BinaryIndex(readInt(tag));
    }
    
    public void endMap(String tag) throws IOException {}
    
}

　　其他的实现还有，cvs文件（CsvInputArchive，CsvOutputArchive）；xml文件（XmlInputArchive，XmlOutputArchive）。