• mongodb批量处理


    mongodb支持批量插入。

    1.使用Java mongodb api

    查看源码com.mongodb.MongoCollectionImpl,有两个方法

    @Override
        public void insertMany(final List<? extends TDocument> documents) {
            insertMany(documents, new InsertManyOptions());
        }
    
        @Override
        public void insertMany(final List<? extends TDocument> documents, final InsertManyOptions options) {
            notNull("documents", documents);
            List<InsertRequest> requests = new ArrayList<InsertRequest>(documents.size());
            for (TDocument document : documents) {
                if (document == null) {
                    throw new IllegalArgumentException("documents can not contain a null value");
                }
                if (getCodec() instanceof CollectibleCodec) {
                    document = ((CollectibleCodec<TDocument>) getCodec()).generateIdIfAbsentFromDocument(document);
                }
                requests.add(new InsertRequest(documentToBsonDocument(document)));
            }
            executor.execute(new MixedBulkWriteOperation(namespace, requests, options.isOrdered(), writeConcern)
                    .bypassDocumentValidation(options.getBypassDocumentValidation()));
        }
    insertMany(final List<? extends TDocument> documents) 默认使用 private boolean ordered = true;即有序插入。
    insertMany(final List<? extends TDocument> documents, final InsertManyOptions options)调用者自己设置。

    ordered属性有什么用?
     /**
         * Gets whether the documents should be inserted in the order provided, stopping on the first failed insertion. The default is true.
         * If false, the server will attempt to insert all the documents regardless of an failures.
         *
         * @return whether the the documents should be inserted in order
         */
        public boolean isOrdered() {
            return ordered;
        }

    大概意思是:

    true:按提供的顺序插入文档,并在首次插入失败时停止。 (按顺序插入文档,遇到失败后停止。之前已经插入成功的不返回,失败及失败之后的不插入。)

    false: 服务器将尝试插入所有文档,而不考虑失败。(只要能插入成功的,都存储进数据库)

    2.spring-data-mongodb

    org.springframework.data.mongodb.core.MongoTemplate

    以下是该方法的源码和使用案例:

        /*
         * (non-Javadoc)
         * @see org.springframework.data.mongodb.core.ExecutableInsertOperation#bulkOps(org.springframework.data.mongodb.core.BulkMode, java.lang.String)
         */
        public BulkOperations bulkOps(BulkMode bulkMode, String collectionName) {
            return bulkOps(bulkMode, null, collectionName);
        }
    public int batchInsertStudents() {
          BulkWriteResult result = null;
          try {
            List<Student> documents = new ArrayList<>();
            String collectionName = "myTestCol";
            BulkOperations bulkOp = this.mongoTemplate.bulkOps(BulkMode.UNORDERED, collectionName);
            for(int i = 502610; i< 2000000; i++) {
                Student student = new Student();
                student.setId(String.valueOf(i));
                student.setAge(i);
                student.setGender("男");
                student.setName("李三"+ i);
                documents.add(student);
            }
            bulkOp.insert(documents);
             result = bulkOp.execute();
          }catch (DuplicateKeyException e) {
              System.out.println("**********" + e.getMessage());
         }
         return result;
      }

    有两种模式:

        /**
         * Mode for bulk operation.
         **/
        enum BulkMode {
    
            /** Perform bulk operations in sequence. The first error will cancel processing. */
            ORDERED,
    
            /** Perform bulk operations in parallel. Processing will continue on errors. */
            UNORDERED
        };

    与之前的order(true|false)相对应。

  • 相关阅读:
    C++11并发——多线程std::thread (一)
    css属性操作
    mustache使用
    layer常用方法代码
    layer使用
    java后台获取和js拼接展示信息
    生成二维码
    循环体中去除一部分特定的数据
    eclipse工具maven项目打包文件不是最新修改的
    sql server数据库备份单个表的结构和数据生成脚本
  • 原文地址:https://www.cnblogs.com/muxi0407/p/11648383.html
Copyright © 2020-2023  润新知