前段时间,线上环境的kafka多集群在采用mirror组件进行跨机房数据同步时,会偶尔出现hang住不稳定的情况:
1. 现象
a. 线上出现返回包序号不一致的现象:"Correlationid for response (13502150) does not match request"而程序hang住,cpu飙高,同步服务停止工作
b. 发生平均频率:线上分3组group×2共6个实例,平均每天2个实例发生
c. 类似线上问题请参考
kafka-mirrormaker问题(https://issues.apache.org/jira/browse/KAFKA-1257)和kafka-producer问题(https://issues.apache.org/jira/browse/KAFKA-4669)
2. 原因
a. kafka网络协议背景
kafka网络协议设计保证连接的请求和响应均是有序的,即对于每个单独的tcp连接下,可以保证发送请求和接收响应包序列均是有序的,同时每个发送请求包和响应包均有唯一递增id关联编号进行关联:“The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well.”出自kafka-network官网介绍;
b. mirrormaker同步判断成功与否逻辑
mirrormaker同步给目标kafka集群的每个数据request包均会搁在本地内存池里,直到收到相同CorrelationId的响应包,然后做两种判断: a. 发送成功,则销毁内存池中的数据请求包,b. 发送失败则数据包放回队列重新进行发送;
c. mirrormaker同步判断线上bug原因
而在判断函数handleCompletedReceives中: 由于条件a,默认认为每个发送请求包和响应包id号是一致的,而并未处理两者id号不一致的异常情况。所以一旦出现id编号不一致异常,则异常一直向上抛,而导致当前"发送请求包"并未得到任何响应处理,同时不会做内存释放最终导致泄露;
d. 目前确定0.8.×、0.9.×版本均会存在线上同样问题
3. 修复方案
修改mirror-maker中kafka-client-0.8.2的源码: 增加出现了错乱包的异常捕获逻辑:把错乱时的数据请求包扔回内存队列进行重发。处理修改源码如下:
/**
* Handle any completed receives and update the response list with the responses received.
* @param responses The list of responses to update
* @param now The current time
*/
private void handleCompletedReceives(List<ClientResponse> responses, long now) {
for (NetworkReceive receive : this.selector.completedReceives()) {
int source = receive.source();
ResponseHeader header = ResponseHeader.parse(receive.payload());
int compared = 0;
ClientRequest req = null;
short apiKey = -1;
do{
req = inFlightRequests.fristSent(source);
if(req == null){
break;
}
apiKey = req.request().header().apiKey();
compared = compareCorrelationId(req.request().header(), header, source);
if (compared < 0 && apiKey != ApiKeys.METADATA.id) {
responses.add(new ClientResponse(req, now, true, null));
}
if (compared < 0 || compared == 0){
req = inFlightRequests.completeNext(source);
}
}while(compared < 0);
if(req == null || compared > 0){
log.error("never go to this line");
continue;
}
Struct body = (Struct) ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
if (apiKey == ApiKeys.METADATA.id) {
handleMetadataResponse(req.request().header(), body, now);
} else {
// need to add body/header to response here
responses.add(new ClientResponse(req, now, false, body));
}
}
}