9.7 dubbo心跳机制

dubbo的心跳机制：

• 目的：检测provider与consumer之间的connection连接是不是还连接着，如果连接断了，需要作出相应的处理。
• 原理：
  • provider：dubbo的心跳默认是在heartbeat（默认是60s）内如果没有接收到消息，就会发送心跳消息，如果连着3次（180s）没有收到心跳响应，provider会关闭channel。
  • consumer：dubbo的心跳默认是在60s内如果没有接收到消息，就会发送心跳消息，如果连着3次（180s）没有收到心跳响应，consumer会进行重连。

来看源码调用链。先看provider端。

一、provider端心跳机制

              -->openServer(URL url)
                 url：dubbo://10.10.10.10:20880/com.alibaba.dubbo.demo.DemoService?anyhost=true&application=demo-provider&bind.ip=10.10.10.10&bind.port=20880&default.server=netty4&dubbo=2.0.0&generic=false&interface=com.alibaba.dubbo.demo.DemoService&methods=sayHello&pid=21999&qos.port=22222&side=provider&timestamp=1520660491836
                -->createServer(URL url)
                    -->HeaderExchanger.bind(URL url, ExchangeHandler handler)
                       url：dubbo://10.10.10.10:20880/com.alibaba.dubbo.demo.DemoService?anyhost=true&application=demo-provider&bind.ip=10.10.10.10&bind.port=20880&channel.readonly.sent=true&codec=dubbo&default.server=netty4&dubbo=2.0.0&generic=false&heartbeat=60000&interface=com.alibaba.dubbo.demo.DemoService&methods=sayHello&pid=21999&qos.port=22222&side=provider&timestamp=1520660491836 handler:DubboProtocol.requestHandler
                      -->new DecodeHandler(new HeaderExchangeHandler(handler)))
                        -->NettyTransporter.bind(URL url, ChannelHandler listener)
                           listener:上边的DecodeHandler实例
                          -->new NettyServer(URL url, ChannelHandler handler)
                            -->ChannelHandler.wrapInternal(ChannelHandler handler, URL url)
                               handler:上边的DecodeHandler实例
                            -->doOpen()//开启netty服务
                      -->new HeaderExchangeServer(Server server)
                         server:上述的NettyServer
                        -->startHeatbeatTimer()

服务端在开启netty服务时， 在调用createServer时，会从url的parameters map中获取heartbeat配置，代码如下：

    private ExchangeServer createServer(URL url) {

        ...

        url = url.addParameterIfAbsent(Constants.HEARTBEAT_KEY, String.valueOf(Constants.DEFAULT_HEARTBEAT));
       
        ...

        ExchangeServer server;
        try {
            server = Exchangers.bind(url, requestHandler);
        } catch (RemotingException e) {
            throw new RpcException("Fail to start server(url: " + url + ") " + e.getMessage(), e);
        }

        ...

        return server;
    }

其中：int DEFAULT_HEARTBEAT = 60 * 1000，即当用户没有配置heartbeat（心跳时间）时，默认heartbeat=60s（即60s内没有接收到任何请求，就会发送心跳信息）。那么这个heartbeat到底该怎么配？

provider端：

    <dubbo:service ...>
        <dubbo:parameter key="heartbeat" value="3000"/>
    </dubbo:service>

consumer端：

    <dubbo:reference ...>
        <dubbo:parameter key="heartbeat" value="3000"/>
    </dubbo:reference>

再来看调用链，当执行到这一句。

ChannelHandler.wrapInternal(ChannelHandler handler, URL url)

会形成一个handler调用链，调用链如下：

MultiMessageHandler
-->handler: HeartbeatHandler
   -->handler: AllChannelHandler
         -->url: providerUrl
         -->executor: FixedExecutor
         -->handler: DecodeHandler
            -->handler: HeaderExchangeHandler
               -->handler: ExchangeHandlerAdapter（DubboProtocol.requestHandler）

这也是netty接收到请求后的处理链路，注意其中有一个HeartbeatHandler。

最后，执行new HeaderExchangeServer(Server server)，来看源码：

public class HeaderExchangeServer implements ExchangeServer {
    /** 心跳定时器 */
    private final ScheduledExecutorService scheduled = Executors.newScheduledThreadPool(1,
            new NamedThreadFactory(
                    "dubbo-remoting-server-heartbeat",
                    true));
    /** NettyServer */
    private final Server server;
    // heartbeat timer
    private ScheduledFuture<?> heatbeatTimer;
    // heartbeat timeout (ms), default value is 0 , won't execute a heartbeat.
    private int heartbeat;
    private int heartbeatTimeout;
    private AtomicBoolean closed = new AtomicBoolean(false);

    public HeaderExchangeServer(Server server) {
        if (server == null) {
            throw new IllegalArgumentException("server == null");
        }
        this.server = server;
        this.heartbeat = server.getUrl().getParameter(Constants.HEARTBEAT_KEY, 0);
        this.heartbeatTimeout = server.getUrl().getParameter(Constants.HEARTBEAT_TIMEOUT_KEY, heartbeat * 3);
        if (heartbeatTimeout < heartbeat * 2) {
            throw new IllegalStateException("heartbeatTimeout < heartbeatInterval * 2");
        }
        startHeatbeatTimer();
    }

    private void startHeatbeatTimer() {
        stopHeartbeatTimer();
        if (heartbeat > 0) {
            heatbeatTimer = scheduled.scheduleWithFixedDelay(
                    new HeartBeatTask(new HeartBeatTask.ChannelProvider() {
                        public Collection<Channel> getChannels() {
                            return Collections.unmodifiableCollection(
                                    HeaderExchangeServer.this.getChannels());
                        }
                    }, heartbeat, heartbeatTimeout),
                    heartbeat, heartbeat, TimeUnit.MILLISECONDS);
        }
    }

    private void stopHeartbeatTimer() {
        try {
            ScheduledFuture<?> timer = heatbeatTimer;
            if (timer != null && !timer.isCancelled()) {
                timer.cancel(true);
            }
        } catch (Throwable t) {
            logger.warn(t.getMessage(), t);
        } finally {
            heatbeatTimer = null;
        }
    }
}

创建HeaderExchangeServer时，初始化了heartbeat（心跳间隔时间）和heartbeatTimeout（心跳响应超时时间：即如果最终发送的心跳在这个时间内都没有返回，则做出响应的处理）。

• heartbeat默认是0（从startHeatbeatTimer()方法可以看出只有heartbeat>0的情况下，才会发心跳，这里heartbeat如果从url的parameter map中获取不到，就是0，但是我们在前边看到dubbo会默认设置heartbeat=60s到parameter map中，所以此处的heartbeat=60s）；
• heartbeatTimeout：默认是heartbeat*3。（原因：假设一端发出一次heartbeatRequest，另一端在heartbeat内没有返回任何响应-包括正常请求响应和心跳响应，此时不能认为是连接断了，因为有可能还是网络抖动什么的导致了tcp包的重传超时等）
• scheduled是一个含有一个线程的定时线程执行器（其中的线程名字为："dubbo-remoting-server-heartbeat-thread-*"）

之后启动心跳定时任务：

• 首先如果原来有心跳定时任务，关闭原来的定时任务
• 之后启动scheduled中的定时线程，从启动该线程开始，每隔heartbeat执行一次HeartBeatTask任务（第一次执行是在启动线程后heartbeat时）

来看一下HeartBeatTask的源码：

final class HeartBeatTask implements Runnable {
    // channel获取器：用于获取所有需要进行心跳检测的channel
    private ChannelProvider channelProvider;
    private int heartbeat;
    private int heartbeatTimeout;

    HeartBeatTask(ChannelProvider provider, int heartbeat, int heartbeatTimeout) {
        this.channelProvider = provider;
        this.heartbeat = heartbeat;
        this.heartbeatTimeout = heartbeatTimeout;
    }

    public void run() {
        try {
            long now = System.currentTimeMillis();
            for (Channel channel : channelProvider.getChannels()) {
                if (channel.isClosed()) {
                    continue;
                }
                try {
                    // 获取最后一次读操作的时间
                    Long lastRead = (Long) channel.getAttribute(
                            HeaderExchangeHandler.KEY_READ_TIMESTAMP);
                    // 获取最后一次写操作的时间
                    Long lastWrite = (Long) channel.getAttribute(
                            HeaderExchangeHandler.KEY_WRITE_TIMESTAMP);
                    // 如果在heartbeat内没有进行读操作或者写操作，则发送心跳请求
                    if ((lastRead != null && now - lastRead > heartbeat)
                            || (lastWrite != null && now - lastWrite > heartbeat)) {
                        Request req = new Request();
                        req.setVersion("2.0.0");
                        req.setTwoWay(true);
                        req.setEvent(Request.HEARTBEAT_EVENT);
                        channel.send(req);
                        if (logger.isDebugEnabled()) {
                            logger.debug("Send heartbeat to remote channel " + channel.getRemoteAddress()
                                    + ", cause: The channel has no data-transmission exceeds a heartbeat period: " + heartbeat + "ms");
                        }
                    }
                    //正常消息和心跳在heartbeatTimeout都没接收到
                    if (lastRead != null && now - lastRead > heartbeatTimeout) {
                        logger.warn("Close channel " + channel
                                + ", because heartbeat read idle time out: " + heartbeatTimeout + "ms");
                        // consumer端进行重连
                        if (channel instanceof Client) {
                            try {
                                ((Client) channel).reconnect();
                            } catch (Exception e) {
                                //do nothing
                            }
                        } else {// provider端关闭连接
                            channel.close();
                        }
                    }
                } catch (Throwable t) {
                    logger.warn("Exception when heartbeat to remote channel " + channel.getRemoteAddress(), t);
                }
            }
        } catch (Throwable t) {
            logger.warn("Unhandled exception when heartbeat, cause: " + t.getMessage(), t);
        }
    }

    interface ChannelProvider {
        Collection<Channel> getChannels();
    }
}            

HeartBeatTask首先获取所有的channelProvider#getChannels获取所有需要心跳检测的channel，channelProvider实例是HeaderExchangeServer中在启动线程定时执行器的时候创建的内部类。

                    new HeartBeatTask.ChannelProvider() {
                        public Collection<Channel> getChannels() {
                            return Collections.unmodifiableCollection(
                                    HeaderExchangeServer.this.getChannels());
                        }
                    }

来看一下HeaderExchangeServer.this.getChannels()：

    public Collection<Channel> getChannels() {
        return (Collection) getExchangeChannels();
    }

    public Collection<ExchangeChannel> getExchangeChannels() {
        Collection<ExchangeChannel> exchangeChannels = new ArrayList<ExchangeChannel>();
        Collection<Channel> channels = server.getChannels();
        if (channels != null && channels.size() > 0) {
            for (Channel channel : channels) {
                exchangeChannels.add(HeaderExchangeChannel.getOrAddChannel(channel));
            }
        }
        return exchangeChannels;
    }

实际上就是获取NettyServer中的全部channel连接。

获取到需要心跳检测的channel后，对每一个channel进行如下判断：

• 如果在heartbeat内没有进行读操作或者写操作，则发送心跳请求
• 如果正常消息和心跳在heartbeatTimeout都没接收到，consumer端会进行重连，provider端会关闭channel

这里比较关键的是lastRead和lastWrite的设置。先来看一下获取：

Long lastRead = (Long) channel.getAttribute(HeaderExchangeHandler.KEY_READ_TIMESTAMP);
Long lastWrite = (Long) channel.getAttribute(HeaderExchangeHandler.KEY_WRITE_TIMESTAMP);

说明有地方在设置这两个值到channel中。

从请求和响应处理来看，无论是请求还是响应都会按照这个顺序处理一遍。

1 MultiMessageHandler
2 -->handler: HeartbeatHandler
3    -->handler: AllChannelHandler
4          -->url: providerUrl
5          -->executor: FixedExecutor
6          -->handler: DecodeHandler
7             -->handler: HeaderExchangeHandler
8                -->handler: ExchangeHandlerAdapter（DubboProtocol.requestHandler）

其中HeartbeatHandler源码如下：

public class HeartbeatHandler extends AbstractChannelHandlerDelegate {

    private static final Logger logger = LoggerFactory.getLogger(HeartbeatHandler.class);

    public static String KEY_READ_TIMESTAMP = "READ_TIMESTAMP";

    public static String KEY_WRITE_TIMESTAMP = "WRITE_TIMESTAMP";

    public HeartbeatHandler(ChannelHandler handler) {
        super(handler);
    }

    public void connected(Channel channel) throws RemotingException {
        setReadTimestamp(channel);
        setWriteTimestamp(channel);
        handler.connected(channel);
    }

    public void disconnected(Channel channel) throws RemotingException {
        clearReadTimestamp(channel);
        clearWriteTimestamp(channel);
        handler.disconnected(channel);
    }

    public void sent(Channel channel, Object message) throws RemotingException {
        setWriteTimestamp(channel);
        handler.sent(channel, message);
    }

    public void received(Channel channel, Object message) throws RemotingException {
        setReadTimestamp(channel);
        if (isHeartbeatRequest(message)) {
            Request req = (Request) message;
            if (req.isTwoWay()) {
                Response res = new Response(req.getId(), req.getVersion());
                res.setEvent(Response.HEARTBEAT_EVENT);
                channel.send(res);
                if (logger.isInfoEnabled()) {
                    int heartbeat = channel.getUrl().getParameter(Constants.HEARTBEAT_KEY, 0);
                    if (logger.isDebugEnabled()) {
                        logger.debug("Received heartbeat from remote channel " + channel.getRemoteAddress()
                                + ", cause: The channel has no data-transmission exceeds a heartbeat period"
                                + (heartbeat > 0 ? ": " + heartbeat + "ms" : ""));
                    }
                }
            }
            return;
        }
        if (isHeartbeatResponse(message)) {
            if (logger.isDebugEnabled()) {
                logger.debug(
                        new StringBuilder(32)
                                .append("Receive heartbeat response in thread ")
                                .append(Thread.currentThread().getName())
                                .toString());
            }
            return;
        }
        handler.received(channel, message);
    }

    private void setReadTimestamp(Channel channel) {
        channel.setAttribute(KEY_READ_TIMESTAMP, System.currentTimeMillis());
    }

    private void setWriteTimestamp(Channel channel) {
        channel.setAttribute(KEY_WRITE_TIMESTAMP, System.currentTimeMillis());
    }

    private void clearReadTimestamp(Channel channel) {
        channel.removeAttribute(KEY_READ_TIMESTAMP);
    }

    private void clearWriteTimestamp(Channel channel) {
        channel.removeAttribute(KEY_WRITE_TIMESTAMP);
    }

    private boolean isHeartbeatRequest(Object message) {
        return message instanceof Request && ((Request) message).isHeartbeat();
    }

    private boolean isHeartbeatResponse(Object message) {
        return message instanceof Response && ((Response) message).isHeartbeat();
    }
}

• 连接完成时：设置lastRead和lastWrite
• 连接断开时：清空lastRead和lastWrite
• 发送消息时：设置lastWrite
• 接收消息时：设置lastRead

之后交由AllChannelHandler进行处理。之后会一直交由HeaderExchangeHandler进行处理。其对lastRead和lastWrite也做了设置和清理：

    public void connected(Channel channel) throws RemotingException {
        channel.setAttribute(KEY_READ_TIMESTAMP, System.currentTimeMillis());
        channel.setAttribute(KEY_WRITE_TIMESTAMP, System.currentTimeMillis());
        ...
    }

    public void disconnected(Channel channel) throws RemotingException {
        channel.setAttribute(KEY_READ_TIMESTAMP, System.currentTimeMillis());
        channel.setAttribute(KEY_WRITE_TIMESTAMP, System.currentTimeMillis());
        ...
    }

    public void sent(Channel channel, Object message) throws RemotingException {
        Throwable exception = null;
        try {
            channel.setAttribute(KEY_WRITE_TIMESTAMP, System.currentTimeMillis());
            ...
        } catch (Throwable t) {
            exception = t;
        }
    }

    public void received(Channel channel, Object message) throws RemotingException {
        channel.setAttribute(KEY_READ_TIMESTAMP, System.currentTimeMillis());
        ...
    }

• 连接完成时：设置lastRead和lastWrite
• 连接断开时：也设置lastRead和lastWrite（为什么？）
• 发送消息时：设置lastWrite
• 接收消息时：设置lastRead

 这里里有个疑问，从handler链来看，无论是请求还是响应都会按照handler链来处理一遍。那么在HeartbeatHandler中已经进行了lastWrite和lastRead的设置，为什么还要在HeaderExchangeHandler中再处理一遍？

最后，provider端认为连接断了，则会关闭channel。来看一下NettyChannel的close方法：

    public void close() {
        // 1 将close属性设为true
        try {
            super.close();
        } catch (Exception e) {
            logger.warn(e.getMessage(), e);
        }
        // 2 从全局NettyChannel缓存器中将当前的NettyChannel删掉
        try {
            removeChannelIfDisconnected(channel);
        } catch (Exception e) {
            logger.warn(e.getMessage(), e);
        }
        // 3 清空当前的NettyChannel中的attributes属性
        try {
            attributes.clear();
        } catch (Exception e) {
            logger.warn(e.getMessage(), e);
        }
        // 4 关闭netty的channel，执行netty的channel的优雅关闭
        try {
            if (logger.isInfoEnabled()) {
                logger.info("Close netty channel " + channel);
            }
            channel.close();
        } catch (Exception e) {
            logger.warn(e.getMessage(), e);
        }
    }

从上边代码来看，假设consumer端挂了，provider端的心跳检测机制可以进行相关的资源回收，所以provider端的心跳检测机制是有必要的。

二、consumer端心跳机制

                      //创建ExchangeClient，对第一次服务发现providers路径下的相关url建立长连接
                      -->getClients(URL url)
                        -->getSharedClient(URL url)
                          -->ExchangeClient exchangeClient = initClient(url)
                            -->Exchangers.connect(url, requestHandler)
                              -->HeaderExchanger.connect(URL url, ExchangeHandler handler)
                                -->new DecodeHandler(new HeaderExchangeHandler(handler)))
                                  -->Transporters.connect(URL url, ChannelHandler... handlers)
                                    -->NettyTransporter.connect(URL url, ChannelHandler listener)
                                      -->new NettyClient(url, listener)
                                        -->new MultiMessageHandler(HeartbeatHandler(AllChannelHandler(handler)))
                                        -->getChannelCodec(url)//获取Codec2，这里是DubboCountCodec实例
                                        -->doOpen()//开启netty客户端
                                        -->doConnect()//连接服务端，建立长连接
                                -->new HeaderExchangeClient(Client client, boolean needHeartbeat)//上述的NettyClient实例，needHeartbeat：true
                                  -->startHeatbeatTimer()//启动心跳计数器

客户端在initClient(url)中设置了heartbeat参数（默认为60s，用户自己设置的方式见“一”中所讲），如下：

    /**
     * Create new connection
     */
    private ExchangeClient initClient(URL url) {
        ...
        // enable heartbeat by default
        url = url.addParameterIfAbsent(Constants.HEARTBEAT_KEY, String.valueOf(Constants.DEFAULT_HEARTBEAT));

        ...

        ExchangeClient client;
        try {
            // connection should be lazy
            if (url.getParameter(Constants.LAZY_CONNECT_KEY, false)) {
                client = new LazyConnectExchangeClient(url, requestHandler);
            } else {
                client = Exchangers.connect(url, requestHandler);
            }
        } catch (RemotingException e) {
            throw new RpcException("Fail to create remoting client for service(" + url + "): " + e.getMessage(), e);
        }
        return client;
    }

与provider类似，来看一下最后开启心跳检测的地方。

public class HeaderExchangeClient implements ExchangeClient {
    private static final ScheduledThreadPoolExecutor scheduled = new ScheduledThreadPoolExecutor(2, new NamedThreadFactory("dubbo-remoting-client-heartbeat", true));
    private final Client client;
    private final ExchangeChannel channel;
    // heartbeat timer
    private ScheduledFuture<?> heartbeatTimer;
    // heartbeat(ms), default value is 0 , won't execute a heartbeat.
    private int heartbeat;
    private int heartbeatTimeout;

    public HeaderExchangeClient(Client client, boolean needHeartbeat) {
        if (client == null) {
            throw new IllegalArgumentException("client == null");
        }
        this.client = client;
        this.channel = new HeaderExchangeChannel(client);
        String dubbo = client.getUrl().getParameter(Constants.DUBBO_VERSION_KEY);
        this.heartbeat = client.getUrl().getParameter(Constants.HEARTBEAT_KEY, dubbo != null && dubbo.startsWith("1.0.") ? Constants.DEFAULT_HEARTBEAT : 0);
        this.heartbeatTimeout = client.getUrl().getParameter(Constants.HEARTBEAT_TIMEOUT_KEY, heartbeat * 3);
        if (heartbeatTimeout < heartbeat * 2) {
            throw new IllegalStateException("heartbeatTimeout < heartbeatInterval * 2");
        }
        if (needHeartbeat) {
            startHeatbeatTimer();
        }
    }

    private void startHeatbeatTimer() {
        stopHeartbeatTimer();
        if (heartbeat > 0) {
            heartbeatTimer = scheduled.scheduleWithFixedDelay(
                    new HeartBeatTask(new HeartBeatTask.ChannelProvider() {
                        public Collection<Channel> getChannels() {
                            return Collections.<Channel>singletonList(HeaderExchangeClient.this);
                        }
                    }, heartbeat, heartbeatTimeout),
                    heartbeat, heartbeat, TimeUnit.MILLISECONDS);
        }
    }

    private void stopHeartbeatTimer() {
        if (heartbeatTimer != null && !heartbeatTimer.isCancelled()) {
            try {
                heartbeatTimer.cancel(true);
                scheduled.purge();
            } catch (Throwable e) {
                if (logger.isWarnEnabled()) {
                    logger.warn(e.getMessage(), e);
                }
            }
        }
        heartbeatTimer = null;
    }
}

主要看一下startHeartbeatTimer()方法，与provider相同，只是provider是获取NettyServer的所有的NettyChannel，而consumer只是获取当前的对象。

consumer的handler处理链与provider完全相同。

最后来看一下consumer的重连机制：AbstractClient#reconnect

    public void reconnect() throws RemotingException {
        disconnect();
        connect();
    }

    public void disconnect() {
        connectLock.lock();
        try {
            destroyConnectStatusCheckCommand();
            try {
                Channel channel = getChannel();
                if (channel != null) {
                    channel.close();
                }
            } catch (Throwable e) {
                logger.warn(e.getMessage(), e);
            }
            try {
                doDisConnect();
            } catch (Throwable e) {
                logger.warn(e.getMessage(), e);
            }
        } finally {
            connectLock.unlock();
        }
    }

    protected void connect() throws RemotingException {
        connectLock.lock();
        try {
            if (isConnected()) {
                return;
            }
            initConnectStatusCheckCommand();
            doConnect();
            if (!isConnected()) {
                throw new RemotingException(this, "Failed connect to server " + getRemoteAddress() + " from " + getClass().getSimpleName() + " "
                        + NetUtils.getLocalHost() + " using dubbo version " + Version.getVersion()
                        + ", cause: Connect wait timeout: " + getTimeout() + "ms.");
            } else {
                if (logger.isInfoEnabled()) {
                    logger.info("Successed connect to server " + getRemoteAddress() + " from " + getClass().getSimpleName() + " "
                            + NetUtils.getLocalHost() + " using dubbo version " + Version.getVersion()
                            + ", channel is " + this.getChannel());
                }
            }
            reconnect_count.set(0);
            reconnect_error_log_flag.set(false);
        } catch (RemotingException e) {
            throw e;
        } catch (Throwable e) {
            throw new RemotingException(this, "Failed connect to server " + getRemoteAddress() + " from " + getClass().getSimpleName() + " "
                    + NetUtils.getLocalHost() + " using dubbo version " + Version.getVersion()
                    + ", cause: " + e.getMessage(), e);
        } finally {
            connectLock.unlock();
        }
    }

代码比较简单，先断连，再连接。

对于心跳机制，netty本身提供了空闲检测：IdleStateHandler。也可以直接基于此实现心跳机制。