前2天刚刚小小的分析下Client端的流程，走的还是比较通顺的，但是RPC的服务端就显然没有那么简单了，毕竟C-S这种模式的，压力和重点都是放在Server端的，所以我也只能做个大概的分析，因为里面细节的东西太多，我也不可能理清所有细节，但是我会集合源代码把主要的流程理理清。如果读者想进一步学习的话，可自行查阅源码。
Server服务端和Client客户端在某些变量的定义上还是一致的，比如服务端也有Call，和Connection，这个很好理解，Call回调，和Connection连接是双向的。首先看一个Server类的定义:
public abstract class Server {
private final boolean authorize;
private boolean isSecurityEnabled;
/**
* The first four bytes of Hadoop RPC connections
* Hadoop RPC的连接魔数字符‘hrpc’
*/
public static final ByteBuffer HEADER = ByteBuffer.wrap("hrpc".getBytes());
// 1 : Introduce ping and server does not throw away RPCs
// 3 : Introduce the protocol into the RPC connection header
// 4 : Introduced SASL security layer
public static final byte CURRENT_VERSION = 4;
.... /** A call queued for handling. */
/** 服务端的Call列表队列 ，与客户端的是不同的*/
private static class Call {
//客户端的Call Id,是从客户端上传过类的
private int id; // the client's call id
//Call回调参数
private Writable param; // the parameter passed
//还保存了与客户端的连接
private Connection connection; // connection to client
//接收到response回应的时间
private long timestamp; // the time received when response is null
// the time served when response is not null
//对于此回调的回应值
private ByteBuffer response; // the response for this call
...... /** Reads calls from a connection and queues them for handling. */
public class Connection {
//连接的RPC头部是否已读
private boolean rpcHeaderRead = false; // if initial rpc header is read
//版本号之后的头部信息是否已读
private boolean headerRead = false; //if the connection header that
//follows version is read.
private SocketChannel channel;
//字节缓冲用于读写回复
private ByteBuffer data;
private ByteBuffer dataLengthBuffer;
//回复Call列表
private LinkedList<Call> responseQueue;
//此连接下的RPC请求数
private volatile int rpcCount = 0; // number of outstanding rpcs
private long lastContact;
private int dataLength;
private Socket socket;
// Cache the remote host & port info so that even if the socket is
// disconnected, we can say where it used to connect to.
private String hostAddress;
private int remotePort;
private InetAddress addr;
..........
volatile private boolean running = true; // true while server runs
//阻塞式Call待处理的队列
private BlockingQueue<Call> callQueue; // queued calls
//与客户端的连接数链表
private List<Connection> connectionList =
Collections.synchronizedList(new LinkedList<Connection>());
//maintain a list
//of client connections
//服务端的监听线程
private Listener listener = null;
//处理应答线程
private Responder responder = null;
private int numConnections = 0;
//处理请求线程组
private Handler[] handlers = null;
.....
好了，下面的分析重点就是3大线程的具体操作了。3大线程的在Server start操作后就会开始工作：
/** Starts the service. Must be called before any calls will be handled. */
/** 服务端的启动方法 */
public synchronized void start() {
//开启3大进程监听线程，回复线程，处理请求线程组
responder.start();
listener.start();
handlers = new Handler[handlerCount];
for (int i = 0; i < handlerCount; i++) {
handlers[i] = new Handler(i);
handlers[i].start();
}
} public Listener() throws IOException {
.....
bind(acceptChannel.socket(), address, backlogLength);
port = acceptChannel.socket().getLocalPort(); //Could be an ephemeral port
// create a selector;
selector= Selector.open();
readers = new Reader[readThreads];
readPool = Executors.newFixedThreadPool(readThreads);
for (int i = 0; i < readThreads; i++) {
Selector readSelector = Selector.open();
Reader reader = new Reader(readSelector);
readers[i] = reader;
//reader Runnable放入线程池中执行
readPool.execute(reader);
}
// Register accepts on the server socket with the selector.
//Java NIO的知识，在selector上注册key的监听事件
acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
this.setName("IPC Server listener on " + port);
this.setDaemon(true);
} @Override
public void run() {
LOG.info(getName() + ": starting");
SERVER.set(Server.this);
while (running) {
SelectionKey key = null;
try {
selector.select();
Iterator<SelectionKey> iter = selector.selectedKeys().iterator();
while (iter.hasNext()) {
key = iter.next();
iter.remove();
try {
if (key.isValid()) {
if (key.isAcceptable())
//Listener的作用就是监听客户端的额连接事件
doAccept(key);
}
} catch (IOException e) {
}
key = null;
..... void doAccept(SelectionKey key) throws IOException, OutOfMemoryError {
Connection c = null;
ServerSocketChannel server = (ServerSocketChannel) key.channel();
SocketChannel channel;
while ((channel = server.accept()) != null) {
channel.configureBlocking(false);
channel.socket().setTcpNoDelay(tcpNoDelay);
Reader reader = getReader();
try {
//连接成功之后，在NIO上注册Read读事件
reader.startAdd();
SelectionKey readKey = reader.registerChannel(channel);
c = new Connection(readKey, channel, System.currentTimeMillis());
readKey.attach(c);
synchronized (connectionList) {
connectionList.add(numConnections, c);
numConnections++;
}
.... public synchronized SelectionKey registerChannel(SocketChannel channel)
throws IOException {
return channel.register(readSelector, SelectionKey.OP_READ);
} public void run() {
LOG.info("Starting SocketReader");
synchronized (this) {
while (running) {
SelectionKey key = null;
try {
readSelector.select();
while (adding) {
this.wait(1000);
}
Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
while (iter.hasNext()) {
key = iter.next();
iter.remove();
if (key.isValid()) {
if (key.isReadable()) {
//Reader的作用就是监听Read读事件
doRead(key);
}
}
key = null;
}
..... void doRead(SelectionKey key) throws InterruptedException {
int count = 0;
Connection c = (Connection)key.attachment();
if (c == null) {
return;
}
c.setLastContact(System.currentTimeMillis());
try {
//监听到RPC请求的读事件后，首先调用下面的方法
count = c.readAndProcess();
.... public int readAndProcess() throws IOException, InterruptedException {
while (true) {
/* Read at most one RPC. If the header is not read completely yet
* then iterate until we read first RPC or until there is no data left.
*/
int count = -1;
//首先读取数据的header头部信息
if (dataLengthBuffer.remaining() > 0) {
count = channelRead(channel, dataLengthBuffer);
if (count < 0 || dataLengthBuffer.remaining() > 0)
return count;
}
if (!rpcHeaderRead) {
//Every connection is expected to send the header.
if (rpcHeaderBuffer == null) {
rpcHeaderBuffer = ByteBuffer.allocate(2);
}
count = channelRead(channel, rpcHeaderBuffer);
if (count < 0 || rpcHeaderBuffer.remaining() > 0) {
return count;
}
//从头部获取版本信息和验证的method类型
int version = rpcHeaderBuffer.get(0);
byte[] method = new byte[] {rpcHeaderBuffer.get(1)};
authMethod = AuthMethod.read(new DataInputStream(
new ByteArrayInputStream(method)));
dataLengthBuffer.flip();
//在这里做if的验证，不符合要求的直接返回
if (!HEADER.equals(dataLengthBuffer) || version != CURRENT_VERSION) {
//Warning is ok since this is not supposed to happen.
LOG.warn("Incorrect header or version mismatch from " +
hostAddress + ":" + remotePort +
" got version " + version +
" expected version " + CURRENT_VERSION);
return -1;
}
....
//继承从channel通道读入数据到data中
count = channelRead(channel, data);
if (data.remaining() == 0) {
dataLengthBuffer.clear();
data.flip();
if (skipInitialSaslHandshake) {
data = null;
skipInitialSaslHandshake = false;
continue;
}
boolean isHeaderRead = headerRead;
//根据是否用了sasl的方式与否进行不同的处理
//SASL是一种用来扩充C/S模式验证能力的机制,我们卡简单的不用这种机制的
if (useSasl) {
saslReadAndProcess(data.array());
} else {
processOneRpc(data.array());
}
..... private void processOneRpc(byte[] buf) throws IOException,
InterruptedException {
if (headerRead) {
//头部信息验证完毕，正式处理处理请求数据
processData(buf);
} else {
//继续验证头部的剩余信息，协议和用户组信息
processHeader(buf);
headerRead = true;
if (!authorizeConnection()) {
throw new AccessControlException("Connection from " + this
+ " for protocol " + header.getProtocol()
+ " is unauthorized for user " + user);
}
}
} private void processData(byte[] buf) throws IOException, InterruptedException {
DataInputStream dis =
new DataInputStream(new ByteArrayInputStream(buf));
int id = dis.readInt(); // try to read an id
if (LOG.isDebugEnabled())
LOG.debug(" got #" + id);
//从配置根据反射获取参数类型
Writable param = ReflectionUtils.newInstance(paramClass, conf);//read param
//数据读入此类似
param.readFields(dis);
//依据ID，和参数构建Server服务的Call回调对象
Call call = new Call(id, param, this);
//放入阻塞式Call队列
callQueue.put(call); // queue the call; maybe blocked here
//增加RPC请求数的数量
incRpcCount(); // Increment the rpc count
} /** 处理请求Call队列 */
private class Handler extends Thread {
public Handler(int instanceNumber) {
this.setDaemon(true);
this.setName("IPC Server handler "+ instanceNumber + " on " + port);
}
@Override
public void run() {
LOG.info(getName() + ": starting");
SERVER.set(Server.this);
ByteArrayOutputStream buf =
new ByteArrayOutputStream(INITIAL_RESP_BUF_SIZE);
//while一直循环处理
while (running) {
try {
//从队列中取出call请求
final Call call = callQueue.take(); // pop the queue; maybe blocked here
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": has #" + call.id + " from " +
call.connection);
String errorClass = null;
String error = null;
Writable value = null;
//设置成当前处理的call请求
CurCall.set(call);
....
CurCall.set(null);
synchronized (call.connection.responseQueue) {
// setupResponse() needs to be sync'ed together with
// responder.doResponse() since setupResponse may use
// SASL to encrypt response data and SASL enforces
// its own message ordering.
//设置回复初始条件
setupResponse(buf, call,
(error == null) ? Status.SUCCESS : Status.ERROR,
value, errorClass, error);
// Discard the large buf and reset it back to
// smaller size to freeup heap
if (buf.size() > maxRespSize) {
LOG.warn("Large response size " + buf.size() + " for call " +
call.toString());
buf = new ByteArrayOutputStream(INITIAL_RESP_BUF_SIZE);
}
//交给responder线程执行写回复操作
responder.doRespond(call);
.... void doRespond(Call call) throws IOException {
synchronized (call.connection.responseQueue) {
call.connection.responseQueue.addLast(call);
if (call.connection.responseQueue.size() == 1) {
processResponse(call.connection.responseQueue, true);
}
}
} private boolean processResponse(LinkedList<Call> responseQueue,
boolean inHandler) throws IOException {
boolean error = true;
boolean done = false; // there is more data for this channel.
int numElements = 0;
Call call = null;
try {
synchronized (responseQueue) {
//
// If there are no items for this channel, then we are done
//
numElements = responseQueue.size();
if (numElements == 0) {
error = false;
return true; // no more data for this channel.
}
//
// Extract the first call
//从Call列表中取出一个做回复
call = responseQueue.removeFirst();
SocketChannel channel = call.connection.channel;
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + ": responding to #" + call.id + " from " +
call.connection);
}
//
// Send as much data as we can in the non-blocking fashion
//向call.response写入回复
int numBytes = channelWrite(channel, call.response);
if (numBytes < 0) {
return true;
}
if (!call.response.hasRemaining()) {
call.connection.decRpcCount();
if (numElements == 1) { // last call fully processes.
done = true; // no more data for this channel.
} else {
done = false; // more calls pending to be sent.
}
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + ": responding to #" + call.id + " from " +
call.connection + " Wrote " + numBytes + " bytes.");
}
} else {
//
// If we were unable to write the entire response out, then
// insert in Selector queue.
//重新把这个call加回call列表
call.connection.responseQueue.addFirst(call);
if (inHandler) {
//inHandler说明此回复将会过会被发送回去，需要改写时间
// set the serve time when the response has to be sent later
//改写Call中收到回复的时间
call.timestamp = System.currentTimeMillis();
incPending();
try {
// Wakeup the thread blocked on select, only then can the call
// to channel.register() complete.
writeSelector.wakeup();
channel.register(writeSelector, SelectionKey.OP_WRITE, call);
..... private int channelWrite(WritableByteChannel channel,
ByteBuffer buffer) throws IOException {
//channel向call.response 的buffer中写入数据
int count = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
channel.write(buffer) : channelIO(null, channel, buffer);
if (count > 0) {
rpcMetrics.incrSentBytes(count);
}
return count;
}//回复Call列表
private LinkedList<Call> responseQueue;
还有一张函数操作的时序图，各个函数的调用流程：
在Hadoop RPC还有一个RPC的辅助类，用来你获取服务端和客户端实例的:
/** Construct a server for a protocol implementation instance listening on a
* port and address, with a secret manager. */
/** 获取服务端的实例 */
public static Server getServer(final Object instance, final String bindAddress, final int port,
final int numHandlers,
final boolean verbose, Configuration conf,
SecretManager<? extends TokenIdentifier> secretManager)
throws IOException {
return new Server(instance, conf, bindAddress, port, numHandlers, verbose, secretManager);
} /**
* Construct & cache an IPC client with the user-provided SocketFactory
* if no cached client exists.
* 获取端缓存中取出客户端，如果没有则创建一个
* @param conf Configuration
* @return an IPC client
*/
private synchronized Client getClient(Configuration conf,
SocketFactory factory) {
// Construct & cache client. The configuration is only used for timeout,
// and Clients have connection pools. So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations. Since the IPC is usually intended globally, not
// per-job, we choose (a).
Client client = clients.get(factory);
if (client == null) {
client = new Client(ObjectWritable.class, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
return client;
}以上就是Hadoop RPC服务端的主要流程分析，确实的忽略了很多细节。整个Hadoop RPC结构是非常复杂的，在Java NIO的基础之上，用了很多动态代理，反射的思想。