Broker读写分离机制
在 RocketMQ 中，有2处地方使用到 "读写分离" 机制。
Broker Master-Slave 读写分离：写操作到 Master Broker，从 Slave Broker 读取消息。Broker 配置为 slaveReadEnable=True（默认False），消息占用内存百度分配置为 accessMessageInMemoryMaxRatio=40（默认）。
Broker Direct Memory-Page Cache 读写：写消息到 Direct Memory（直接内存，简称 DM），从操作系统的 Page Cache 中读取消息。Master Broker 配置读写分离开关为 tranientStorePoolEnable=True（默认为False），写入 DM 存储数量，配置 trainsientStorePoolSize 至少大于0（默认为5，建议不修改），刷盘类型配置为 flushDiskType=FlushDiskType.ASYNC_FLUSH，即异步输盘。
首先我们来讲 Master-Slave 读写分离机制。通常，都是 Master 提供读写处理，如果 Master 负载较高，就从 Slave 读取，整个过程如下：
该机制的实现分为以下两个步骤。
第一步：Broker 在处理 Pull 消息时，计算下次是否从 Slave 拉取消息，是通过 org.apache.rocketmq.store.DefaultMessageStore.getMessage() 方法实现的，代码路径：D: ocketmq-masterstoresrcmainjavaorgapache ocketmqstoreDefaultMessageStore.java，代码如下：
1 SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset); 2 if (bufferConsumeQueue != null) { 3 try { 4 status = GetMessageStatus.NO_MATCHED_MESSAGE; 5 6 long nextPhyFileStartOffset = Long.MIN_VALUE; 7 long maxPhyOffsetPulling = 0; #表示拉取的最大消息位点。 8 9 int i = 0; 10 final int maxFilterMessageCount = Math.max(16000, maxMsgNums * ConsumeQueue.CQ_STORE_UNIT_SIZE); 11 final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded(); 12 ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit(); 13 for (; i < bufferConsumeQueue.getSize() && i < maxFilterMessageCount; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) { 14 long offsetPy = bufferConsumeQueue.getByteBuffer().getLong(); 15 int sizePy = bufferConsumeQueue.getByteBuffer().getInt(); 16 long tagsCode = bufferConsumeQueue.getByteBuffer().getLong(); 17 18 maxPhyOffsetPulling = offsetPy; 19 20 if (nextPhyFileStartOffset != Long.MIN_VALUE) { 21 if (offsetPy < nextPhyFileStartOffset) 22 continue; 23 } 24 25 boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy); #表示当前 Master Broker 存储的所有消息的最大物理位点 26 27 if (this.isTheBatchFull(sizePy, maxMsgNums, getResult.getBufferTotalSize(), getResult.getMessageCount(), 28 isInDisk)) { 29 break; 30 } 31 32 boolean extRet = false, isTagsCodeLegal = true; 33 if (consumeQueue.isExtAddr(tagsCode)) { 34 extRet = consumeQueue.getExt(tagsCode, cqExtUnit); 35 if (extRet) { 36 tagsCode = cqExtUnit.getTagsCode(); 37 } else { 38 // can't find ext content.Client will filter messages by tag also. 39 log.error("[BUG] can't find consume queue extend file content!addr={}, offsetPy={}, sizePy={}, topic={}, group={}", 40 tagsCode, offsetPy, sizePy, topic, group); 41 isTagsCodeLegal = false; 42 } 43 } 44 45 if (messageFilter != null 46 && !messageFilter.isMatchedByConsumeQueue(isTagsCodeLegal ? tagsCode : null, extRet ? cqExtUnit : null)) { 47 if (getResult.getBufferTotalSize() == 0) { 48 status = GetMessageStatus.NO_MATCHED_MESSAGE; 49 } 50 51 continue; 52 } 53 54 SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy); 55 if (null == selectResult) { 56 if (getResult.getBufferTotalSize() == 0) { 57 status = GetMessageStatus.MESSAGE_WAS_REMOVING; 58 } 59 60 nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy); 61 continue; 62 } 63 64 if (messageFilter != null 65 && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) { 66 if (getResult.getBufferTotalSize() == 0) { 67 status = GetMessageStatus.NO_MATCHED_MESSAGE; 68 } 69 // release... 70 selectResult.release(); 71 continue; 72 } 73 74 this.storeStatsService.getGetMessageTransferedMsgCount().incrementAndGet(); 75 getResult.addMessage(selectResult); 76 status = GetMessageStatus.FOUND; 77 nextPhyFileStartOffset = Long.MIN_VALUE; 78 } 79 80 if (diskFallRecorded) { 81 long fallBehind = maxOffsetPy - maxPhyOffsetPulling; 82 brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind); 83 } 84 85 nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE); 86 87 long diff = maxOffsetPy - maxPhyOffsetPulling; #diff：是 maxOffsetPy 和 maxPhyOffsetPulling 两者的差值，表示还有多少消息没有拉取 88 long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE #StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE：表示当前 Master Broker 全部的物理内存大小。
#memory：Broker 认为可使用最大内存，该值可以通过 accessMessageInMemoryMaxRatio 配置项决定，默认 accessMessageInMemoryMaxRatio=40，如果物理内存为 100MB，那么memory=40MB。 89 * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
#设置下次从 Master 或 Slave 拉取消息 90 getResult.setSuggestPullingFromSlave(diff > memory);#表示没有拉取的消息比分配的内存大，如果diff>memory的值为True，则说明此事Master Broker内存繁忙，该从slave拉 91 } finally { 92 93 bufferConsumeQueue.release(); 94 } 95 }
第二步：通知客户端下次从哪个 Broker 拉取消息。在消费者 Pull 消息返回结果时，根据第一步设置的 suggestPullingFromSlave 值返回给消费者，该过程通过 D: ocketmq-masterrokersrcmainjavaorgapache ocketmqrokerprocessorPullMessageProcessor.java 中 processRequest()方法实现，具体代码如下：
1 private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend) 2 throws RemotingCommandException { 3 RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class); 4 final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader(); 5 final PullMessageRequestHeader requestHeader = 6 (PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class); 7 8 response.setOpaque(request.getOpaque()); 9 10 log.debug("receive PullMessage request command, {}", request); 11 12 if (!PermName.isReadable(this.brokerController.getBrokerConfig().getBrokerPermission())) { 13 response.setCode(ResponseCode.NO_PERMISSION); 14 response.setRemark(String.format("the broker[%s] pulling message is forbidden", this.brokerController.getBrokerConfig().getBrokerIP1())); 15 return response; 16 } 17 18 SubscriptionGroupConfig subscriptionGroupConfig = 19 this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getConsumerGroup()); 20 if (null == subscriptionGroupConfig) { 21 response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST); 22 response.setRemark(String.format("subscription group [%s] does not exist, %s", requestHeader.getConsumerGroup(), FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST))); 23 return response; 24 } 25 26 if (!subscriptionGroupConfig.isConsumeEnable()) { 27 response.setCode(ResponseCode.NO_PERMISSION); 28 response.setRemark("subscription group no permission, " + requestHeader.getConsumerGroup()); 29 return response; 30 } 31 32 final boolean hasSuspendFlag = PullSysFlag.hasSuspendFlag(requestHeader.getSysFlag()); 33 final boolean hasCommitOffsetFlag = PullSysFlag.hasCommitOffsetFlag(requestHeader.getSysFlag()); 34 final boolean hasSubscriptionFlag = PullSysFlag.hasSubscriptionFlag(requestHeader.getSysFlag()); 35 36 final long suspendTimeoutMillisLong = hasSuspendFlag ? requestHeader.getSuspendTimeoutMillis() : 0; 37 38 TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic()); 39 if (null == topicConfig) { 40 log.error("the topic {} not exist, consumer: {}", requestHeader.getTopic(), RemotingHelper.parseChannelRemoteAddr(channel)); 41 response.setCode(ResponseCode.TOPIC_NOT_EXIST); 42 response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(), FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL))); 43 return response; 44 } 45 46 if (!PermName.isReadable(topicConfig.getPerm())) { 47 response.setCode(ResponseCode.NO_PERMISSION); 48 response.setRemark("the topic[" + requestHeader.getTopic() + "] pulling message is forbidden"); 49 return response; 50 } 51 52 if (requestHeader.getQueueId() < 0 || requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) { 53 String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] consumer:[%s]", 54 requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(), channel.remoteAddress()); 55 log.warn(errorInfo); 56 response.setCode(ResponseCode.SYSTEM_ERROR); 57 response.setRemark(errorInfo); 58 return response; 59 } 60 61 SubscriptionData subscriptionData = null; 62 ConsumerFilterData consumerFilterData = null; 63 if (hasSubscriptionFlag) { 64 try { 65 subscriptionData = FilterAPI.build( 66 requestHeader.getTopic(), requestHeader.getSubscription(), requestHeader.getExpressionType() 67 ); 68 if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) { 69 consumerFilterData = ConsumerFilterManager.build( 70 requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getSubscription(), 71 requestHeader.getExpressionType(), requestHeader.getSubVersion() 72 ); 73 assert consumerFilterData != null; 74 } 75 } catch (Exception e) { 76 log.warn("Parse the consumer's subscription[{}] failed, group: {}", requestHeader.getSubscription(), 77 requestHeader.getConsumerGroup()); 78 response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED); 79 response.setRemark("parse the consumer's subscription failed"); 80 return response; 81 } 82 } else { 83 ConsumerGroupInfo consumerGroupInfo = 84 this.brokerController.getConsumerManager().getConsumerGroupInfo(requestHeader.getConsumerGroup()); 85 if (null == consumerGroupInfo) { 86 log.warn("the consumer's group info not exist, group: {}", requestHeader.getConsumerGroup()); 87 response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST); 88 response.setRemark("the consumer's group info not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC)); 89 return response; 90 } 91 92 if (!subscriptionGroupConfig.isConsumeBroadcastEnable() 93 && consumerGroupInfo.getMessageModel() == MessageModel.BROADCASTING) { 94 response.setCode(ResponseCode.NO_PERMISSION); 95 response.setRemark("the consumer group[" + requestHeader.getConsumerGroup() + "] can not consume by broadcast way"); 96 return response; 97 } 98 99 subscriptionData = consumerGroupInfo.findSubscriptionData(requestHeader.getTopic()); 100 if (null == subscriptionData) { 101 log.warn("the consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), requestHeader.getTopic()); 102 response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST); 103 response.setRemark("the consumer's subscription not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC)); 104 return response; 105 } 106 107 if (subscriptionData.getSubVersion() < requestHeader.getSubVersion()) { 108 log.warn("The broker's subscription is not latest, group: {} {}", requestHeader.getConsumerGroup(), 109 subscriptionData.getSubString()); 110 response.setCode(ResponseCode.SUBSCRIPTION_NOT_LATEST); 111 response.setRemark("the consumer's subscription not latest"); 112 return response; 113 } 114 if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) { 115 consumerFilterData = this.brokerController.getConsumerFilterManager().get(requestHeader.getTopic(), 116 requestHeader.getConsumerGroup()); 117 if (consumerFilterData == null) { 118 response.setCode(ResponseCode.FILTER_DATA_NOT_EXIST); 119 response.setRemark("The broker's consumer filter data is not exist!Your expression may be wrong!"); 120 return response; 121 } 122 if (consumerFilterData.getClientVersion() < requestHeader.getSubVersion()) { 123 log.warn("The broker's consumer filter data is not latest, group: {}, topic: {}, serverV: {}, clientV: {}", 124 requestHeader.getConsumerGroup(), requestHeader.getTopic(), consumerFilterData.getClientVersion(), requestHeader.getSubVersion()); 125 response.setCode(ResponseCode.FILTER_DATA_NOT_LATEST); 126 response.setRemark("the consumer's consumer filter data not latest"); 127 return response; 128 } 129 } 130 } 131 132 if (!ExpressionType.isTagType(subscriptionData.getExpressionType()) 133 && !this.brokerController.getBrokerConfig().isEnablePropertyFilter()) { 134 response.setCode(ResponseCode.SYSTEM_ERROR); 135 response.setRemark("The broker does not support consumer to filter message by " + subscriptionData.getExpressionType()); 136 return response; 137 } 138 139 MessageFilter messageFilter; 140 if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) { 141 messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData, 142 this.brokerController.getConsumerFilterManager()); 143 } else { 144 messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData, 145 this.brokerController.getConsumerFilterManager()); 146 } 147 148 final GetMessageResult getMessageResult = 149 this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(), 150 requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter); 151 if (getMessageResult != null) { 152 response.setRemark(getMessageResult.getStatus().name()); 153 responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset()); 154 responseHeader.setMinOffset(getMessageResult.getMinOffset()); 155 responseHeader.setMaxOffset(getMessageResult.getMaxOffset()); 156 157 if (getMessageResult.isSuggestPullingFromSlave()) { 158 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly()); 159 } else { 160 responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID); 161 } 162 163 switch (this.brokerController.getMessageStoreConfig().getBrokerRole()) { 164 case ASYNC_MASTER: 165 case SYNC_MASTER: 166 break; 167 case SLAVE: 168 if (!this.brokerController.getBrokerConfig().isSlaveReadEnable()) { 169 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 170 responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID); 171 } 172 break; 173 } 174 # slave 读取开关配置 175 if (this.brokerController.getBrokerConfig().isSlaveReadEnable()) { 176 // consume too slow ,redirect to another machine 177 if (getMessageResult.isSuggestPullingFromSlave()) { 178 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly()); 179 } 180 // consume ok 181 else { 182 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId()); 183 } 184 } else { 185 responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID); 186 } 187 188 switch (getMessageResult.getStatus()) { 189 case FOUND: 190 response.setCode(ResponseCode.SUCCESS); 191 break; 192 case MESSAGE_WAS_REMOVING: 193 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 194 break; 195 case NO_MATCHED_LOGIC_QUEUE: 196 case NO_MESSAGE_IN_QUEUE: 197 if (0 != requestHeader.getQueueOffset()) { 198 response.setCode(ResponseCode.PULL_OFFSET_MOVED); 199 200 // XXX: warn and notify me 201 log.info("the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}", 202 requestHeader.getQueueOffset(), 203 getMessageResult.getNextBeginOffset(), 204 requestHeader.getTopic(), 205 requestHeader.getQueueId(), 206 requestHeader.getConsumerGroup() 207 ); 208 } else { 209 response.setCode(ResponseCode.PULL_NOT_FOUND); 210 } 211 break; 212 case NO_MATCHED_MESSAGE: 213 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 214 break; 215 case OFFSET_FOUND_NULL: 216 response.setCode(ResponseCode.PULL_NOT_FOUND); 217 break; 218 case OFFSET_OVERFLOW_BADLY: 219 response.setCode(ResponseCode.PULL_OFFSET_MOVED); 220 // XXX: warn and notify me 221 log.info("the request offset: {} over flow badly, broker max offset: {}, consumer: {}", 222 requestHeader.getQueueOffset(), getMessageResult.getMaxOffset(), channel.remoteAddress()); 223 break; 224 case OFFSET_OVERFLOW_ONE: 225 response.setCode(ResponseCode.PULL_NOT_FOUND); 226 break; 227 case OFFSET_TOO_SMALL: 228 response.setCode(ResponseCode.PULL_OFFSET_MOVED); 229 log.info("the request offset too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}", 230 requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(), 231 getMessageResult.getMinOffset(), channel.remoteAddress()); 232 break; 233 default: 234 assert false; 235 break; 236 } 237 238 if (this.hasConsumeMessageHook()) { 239 ConsumeMessageContext context = new ConsumeMessageContext(); 240 context.setConsumerGroup(requestHeader.getConsumerGroup()); 241 context.setTopic(requestHeader.getTopic()); 242 context.setQueueId(requestHeader.getQueueId()); 243 244 String owner = request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER); 245 246 switch (response.getCode()) { 247 case ResponseCode.SUCCESS: 248 int commercialBaseCount = brokerController.getBrokerConfig().getCommercialBaseCount(); 249 int incValue = getMessageResult.getMsgCount4Commercial() * commercialBaseCount; 250 251 context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_SUCCESS); 252 context.setCommercialRcvTimes(incValue); 253 context.setCommercialRcvSize(getMessageResult.getBufferTotalSize()); 254 context.setCommercialOwner(owner); 255 256 break; 257 case ResponseCode.PULL_NOT_FOUND: 258 if (!brokerAllowSuspend) { 259 260 context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS); 261 context.setCommercialRcvTimes(1); 262 context.setCommercialOwner(owner); 263 264 } 265 break; 266 case ResponseCode.PULL_RETRY_IMMEDIATELY: 267 case ResponseCode.PULL_OFFSET_MOVED: 268 context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS); 269 context.setCommercialRcvTimes(1); 270 context.setCommercialOwner(owner); 271 break; 272 default: 273 assert false; 274 break; 275 } 276 277 this.executeConsumeMessageHookBefore(context); 278 } 279 280 switch (response.getCode()) { 281 case ResponseCode.SUCCESS: 282 283 this.brokerController.getBrokerStatsManager().incGroupGetNums(requestHeader.getConsumerGroup(), requestHeader.getTopic(), 284 getMessageResult.getMessageCount()); 285 286 this.brokerController.getBrokerStatsManager().incGroupGetSize(requestHeader.getConsumerGroup(), requestHeader.getTopic(), 287 getMessageResult.getBufferTotalSize()); 288 289 this.brokerController.getBrokerStatsManager().incBrokerGetNums(getMessageResult.getMessageCount()); 290 if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) { 291 final long beginTimeMills = this.brokerController.getMessageStore().now(); 292 final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId()); 293 this.brokerController.getBrokerStatsManager().incGroupGetLatency(requestHeader.getConsumerGroup(), 294 requestHeader.getTopic(), requestHeader.getQueueId(), 295 (int) (this.brokerController.getMessageStore().now() - beginTimeMills)); 296 response.setBody(r); 297 } else { 298 try { 299 FileRegion fileRegion = 300 new ManyMessageTransfer(response.encodeHeader(getMessageResult.getBufferTotalSize()), getMessageResult); 301 channel.writeAndFlush(fileRegion).addListener(new ChannelFutureListener() { 302 @Override 303 public void operationComplete(ChannelFuture future) throws Exception { 304 getMessageResult.release(); 305 if (!future.isSuccess()) { 306 log.error("transfer many message by pagecache failed, {}", channel.remoteAddress(), future.cause()); 307 } 308 } 309 }); 310 } catch (Throwable e) { 311 log.error("transfer many message by pagecache exception", e); 312 getMessageResult.release(); 313 } 314 315 response = null; 316 } 317 break; 318 case ResponseCode.PULL_NOT_FOUND: 319 320 if (brokerAllowSuspend && hasSuspendFlag) { 321 long pollingTimeMills = suspendTimeoutMillisLong; 322 if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) { 323 pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills(); 324 } 325 326 String topic = requestHeader.getTopic(); 327 long offset = requestHeader.getQueueOffset(); 328 int queueId = requestHeader.getQueueId(); 329 PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills, 330 this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter); 331 this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest); 332 response = null; 333 break; 334 } 335 336 case ResponseCode.PULL_RETRY_IMMEDIATELY: 337 break; 338 case ResponseCode.PULL_OFFSET_MOVED: 339 if (this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE 340 || this.brokerController.getMessageStoreConfig().isOffsetCheckInSlave()) { 341 MessageQueue mq = new MessageQueue(); 342 mq.setTopic(requestHeader.getTopic()); 343 mq.setQueueId(requestHeader.getQueueId()); 344 mq.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName()); 345 346 OffsetMovedEvent event = new OffsetMovedEvent(); 347 event.setConsumerGroup(requestHeader.getConsumerGroup()); 348 event.setMessageQueue(mq); 349 event.setOffsetRequest(requestHeader.getQueueOffset()); 350 event.setOffsetNew(getMessageResult.getNextBeginOffset()); 351 this.generateOffsetMovedEvent(event); 352 log.warn( 353 "PULL_OFFSET_MOVED:correction offset. topic={}, groupId={}, requestOffset={}, newOffset={}, suggestBrokerId={}", 354 requestHeader.getTopic(), requestHeader.getConsumerGroup(), event.getOffsetRequest(), event.getOffsetNew(), 355 responseHeader.getSuggestWhichBrokerId()); 356 } else { 357 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId()); 358 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY); 359 log.warn("PULL_OFFSET_MOVED:none correction. topic={}, groupId={}, requestOffset={}, suggestBrokerId={}", 360 requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getQueueOffset(), 361 responseHeader.getSuggestWhichBrokerId()); 362 } 363 364 break; 365 default: 366 assert false; 367 } 368 } else { 369 response.setCode(ResponseCode.SYSTEM_ERROR); 370 response.setRemark("store getMessage return null"); 371 } 372 373 boolean storeOffsetEnable = brokerAllowSuspend; 374 storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag; 375 storeOffsetEnable = storeOffsetEnable 376 && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE; 377 if (storeOffsetEnable) { 378 this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel), 379 requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset()); 380 } 381 return response; 382 }
通过查看以上代码第168行，我们知道要想从 Slave 读取消息，需要设置 slaveReadEnable=True，此时会根据第一步返回的 suggestPullingFromSlave 值告诉客户端下次可以从哪个 Broker 拉取消息。suggestPullingFromSlave=1 表示从 Slave 拉取，suggestPullingFromSlave=0 表示从 Master 拉取。
在了解了 Master-Slave 读写分离机制后，接着讲 Direct Memory-Page Cache 的读写分离机制：
以上逻辑通过 D: ocketmq-masterstoresrcmainjavaorgapache ocketmqstoreMappedFile.java 中 appendMessagesInner()，具体代码如下：
1 public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) { 2 assert messageExt != null; 3 assert cb != null; 4 5 int currentPos = this.wrotePosition.get(); 6 7 if (currentPos < this.fileSize) { 8 ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice(); 9 byteBuffer.position(currentPos); 10 AppendMessageResult result; 11 if (messageExt instanceof MessageExtBrokerInner) { 12 result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt); 13 } else if (messageExt instanceof MessageExtBatch) { 14 result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBatch) messageExt); 15 } else { 16 return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR); 17 } 18 this.wrotePosition.addAndGet(result.getWroteBytes()); 19 this.storeTimestamp = result.getStoreTimestamp(); 20 return result; 21 } 22 log.error("MappedFile.appendMessage return null, wrotePosition: {} fileSize: {}", currentPos, this.fileSize); 23 return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR); 24 }
这段代码中，writeBuffer 表示 从 DM 中申请的缓存；mappendByteBuffer 表示从 Page Cache 中申请的缓存。如果 Broker 设置 transientStorePoolEnable=true，并且异步刷盘，则存储层srcmainjavaorgapache ocketmqstoreDefaultMessageStore.java 在初始化时会调用 TransientStorePool.init() 方法（按照配置的 Buffer 个数）初始化 writeBuffer。代码如下：
1 public DefaultMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager, 2 final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException { 3 this.messageArrivingListener = messageArrivingListener; 4 this.brokerConfig = brokerConfig; 5 this.messageStoreConfig = messageStoreConfig; 6 this.brokerStatsManager = brokerStatsManager; 7 this.allocateMappedFileService = new AllocateMappedFileService(this); 8 if (messageStoreConfig.isEnableDLegerCommitLog()) { 9 this.commitLog = new DLedgerCommitLog(this); 10 } else { 11 this.commitLog = new CommitLog(this); 12 } 13 this.consumeQueueTable = new ConcurrentHashMap<>(32); 14 15 this.flushConsumeQueueService = new FlushConsumeQueueService(); 16 this.cleanCommitLogService = new CleanCommitLogService(); 17 this.cleanConsumeQueueService = new CleanConsumeQueueService(); 18 this.storeStatsService = new StoreStatsService(); 19 this.indexService = new IndexService(this); 20 if (!messageStoreConfig.isEnableDLegerCommitLog()) { 21 this.haService = new HAService(this); 22 } else { 23 this.haService = null; 24 } 25 this.reputMessageService = new ReputMessageService(); 26 27 this.scheduleMessageService = new ScheduleMessageService(this); 28 29 this.transientStorePool = new TransientStorePool(messageStoreConfig); 30 31 if (messageStoreConfig.isTransientStorePoolEnable()) { 32 this.transientStorePool.init(); 33 } 34 35 this.allocateMappedFileService.start(); 36 37 this.indexService.start(); 38 39 this.dispatcherList = new LinkedList<>(); 40 this.dispatcherList.addLast(new CommitLogDispatcherBuildConsumeQueue()); 41 this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex()); 42 43 File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir())); 44 MappedFile.ensureDirOK(file.getParent()); 45 lockFile = new RandomAccessFile(file, "rw"); 46 }
初始化 writeBuffer 后，当生产者将消息发送到 Broker时，Broker 将消息写入 writeBuffer，然后被异步转存服务不断地从 DM 中 Commit 到 Page Cache 中。消费者此时从哪儿读取数据呢？消费者拉取消息的实现在 D: ocketmq-masterstoresrcmainjavaorgapache ocketmqstoreMappedFile.java 中 selectMappedBuffer() 方法中，具体代码如下：
1 if (this.hold()) { 2 ByteBuffer byteBuffer = this.mappedByteBuffer.slice(); 3 byteBuffer.position(pos); 4 ByteBuffer byteBufferNew = byteBuffer.slice(); 5 byteBufferNew.limit(size); 6 return new SelectMappedBufferResult(this.fileFromOffset + pos, byteBufferNew, size, this);
从代码中可以看到，消费者始终从 mappedByteBuffer（即 Page Cache）读取消息。
读写分离能够最大限度地提供吞吐量，同时会增加数据不一致性的风险，建议生产环境谨慎使用。