在使用mongo的聚合管道的时候会发现如果涉及的文档数过多,出现如下报错
com.mongodb.MongoCommandException: Command failed with error 146 (ExceededMemoryLimit): '$push used too much memory and cannot spill to disk. Memory limit: 104857600 bytes' on server 192.168.2.234:27017. The full response is {"ok": 0.0, "errmsg": "$push used too much memory and cannot spill to disk. Memory limit: 104857600 bytes", "code": 146, "codeName": "ExceededMemoryLimit"} at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:175) at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:302) at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:258) at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:500) at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71) at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:224) at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:202) at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:110) at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343) at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334) at com.mongodb.internal.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220) at com.mongodb.internal.operation.CommandOperationHelper$5.call(CommandOperationHelper.java:206) at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:462) at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:203) at com.mongodb.internal.operation.AggregateOperationImpl.execute(AggregateOperationImpl.java:189) at com.mongodb.internal.operation.AggregateOperation.execute(AggregateOperation.java:296) at com.mongodb.internal.operation.AggregateOperation.execute(AggregateOperation.java:41) at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:190) at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:135) at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:92) at com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:121) at com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:130)
经过网上查询,源赖式,佐田,mongo把这两个聚合操作里面的内存限制在了100M,具体不太清楚为啥限制,默认是100M吧,但是在之后mongodb版本里面可以进行设置,通过mongo shell如下操作即可,命令行的话,通过setParameter
> newLimit = 100 * 1024 * 1024;> db.adminCommand({setParameter: 1, internalQueryMaxAddToSetBytes: newLimit})
不过,不太建议改这个东西,没有意义,因为mongo的文档大小有限制,为16M,就算你在聚合管道的某个阶段给$push操作数组的内存
大小放开,但是,最后如果输出的文档包含此过大数组,还是会报错
com.mongodb.MongoCommandException: Command failed with error 10334 (BSONObjectTooLarge): 'BSONObj size: 23537822 (0x167289E) is invalid. Size must be between 0 and 16793600(16MB) First element:
后面太长了,就不贴了,所以一般改了也没啥用吧,除非确实是中间结果需要处理。
参考链接:
https://stackoverflow.com/questions/63498668/how-to-set-internalquerymaxaddtosetbytes-in-mongo-db-version-3-6-9