• pringBoot-MongoDB 索引冲突分析及解决【华为云技术分享】


    版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
    本文链接:https://blog.csdn.net/devcloud/article/details/100119386

    一、背景

    spring-data-mongo 实现了基于 MongoDB 的 ORM-Mapping 能力,
    通过一些简单的注解、Query封装以及工具类,就可以通过对象操作来实现集合、文档的增删改查;
    在 SpringBoot 体系中,spring-data-mongo 是 MongoDB Java 工具库的不二之选。

    二、问题产生

    在一次项目问题的追踪中,发现SpringBoot 应用启动失败,报错信息如下:

     1 Error creating bean with name 'mongoTemplate' defined in class path resource [org/bootfoo/BootConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.mongodb.core.MongoTemplate]: Factory method 'mongoTemplate' threw exception; nested exception is org.springframework.dao.DataIntegrityViolationException: Cannot create index for 'deviceId' in collection 'T_MDevice' with keys '{ "deviceId" : 1}' and options '{ "name" : "deviceId"}'. Index already defined as '{ "v" : 1 , "unique" : true , "key" : { "deviceId" : 1} , "name" : "deviceId" , "ns" : "appdb.T_MDevice"}'.; nested exception is com.mongodb.MongoCommandException: Command failed with error 85: 'exception: Index with name: deviceId already exists with different options' on server 127.0.0.1:27017. The full response is { "createdCollectionAutomatically" : false, "numIndexesBefore" : 6, "errmsg" : "exception: Index with name: deviceId already exists with different options", "code" : 85, "ok" : 0.0 }
     2     at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
     3     at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
     4     at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
     5     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
     6     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
     7 
     8 ...
     9 
    10 Caused by: org.springframework.dao.DataIntegrityViolationException: Cannot create index for 'deviceId' in collection 'T_MDevice' with keys '{ "deviceId" : 1}' and options '{ "name" : "deviceId"}'. Index already defined as '{ "v" : 1 , "unique" : true , "key" : { "deviceId" : 1} , "name" : "deviceId" , "ns" : "appdb.T_MDevice"}'.; nested exception is com.mongodb.MongoCommandException: Command failed with error 85: 'exception: Index with name: deviceId already exists with different options' on server 127.0.0.1:27017. The full response is { "createdCollectionAutomatically" : false, "numIndexesBefore" : 6, "errmsg" : "exception: Index with name: deviceId already exists with different options", "code" : 85, "ok" : 0.0 }
    11     at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.createIndex(MongoPersistentEntityIndexCreator.java:157)
    12     at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForAndCreateIndexes(MongoPersistentEntityIndexCreator.java:133)
    13     at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForIndexes(MongoPersistentEntityIndexCreator.java:125)
    14     at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.<init>(MongoPersistentEntityIndexCreator.java:91)
    15     at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.<init>(MongoPersistentEntityIndexCreator.java:68)
    16     at org.springframework.data.mongodb.core.MongoTemplate.<init>(MongoTemplate.java:229)
    17     at org.bootfoo.BootConfiguration.mongoTemplate(BootConfiguration.java:121)
    18     at org.bootfoo.BootConfiguration$$EnhancerBySpringCGLIB$$1963a75.CGLIB$mongoTemplate$2(<generated>)
    19     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    20     at java.lang.reflect.Method.invoke(Unknown Source)
    21     at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    22     ... 58 more
    23 
    24 Caused by: com.mongodb.MongoCommandException: Command failed with error 85: 'exception: Index with name: deviceId already exists with different options' on server 127.0.0.1:27017. The full response is { "createdCollectionAutomatically" : false, "numIndexesBefore" : 6, "errmsg" : "exception: Index with name: deviceId already exists with different options", "code" : 85, "ok" : 0.0 }
    25     at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
    26     at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
    27     at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)

    关键信息:org.springframework.dao.DataIntegrityViolationException: Cannot create index

    从异常信息上看,出现的是索引冲突(Command failed with error 85),spring-data-mongo 组件在程序启动时会实现根据注解创建索引的功能。
    查看业务实体定义:

    1 @Document(collection = "T_MDevice")
    2 public class MDevice {
    3 
    4     @Id
    5     private String id;
    6 
    7     @Indexed(unique=true)
    8     private String deviceId;

    deviceId 这个字段上定义了一个索引,unique=true表示这是一个唯一索引。
    我们继续 查看 MongoDB中表的定义:

     1 db.getCollection('T_MDevice').getIndexes()
     2 
     3 >>
     4 [
     5     {
     6         "v" : 1,
     7         "key" : {
     8             "_id" : 1
     9         },
    10         "name" : "_id_",
    11         "ns" : "appdb.T_MDevice"
    12     },
    13     {
    14         "v" : 1,
    15         "key" : {
    16             "deviceId" : 1
    17         },
    18         "name" : "deviceId",
    19         "ns" : "appdb.T_MDevice"
    20     }
    21 ]

    发现数据库表中同样存在一个名为 deviceId的索引,但是并非唯一索引!

    三、详细分析

    为了核实错误产生的原因,我们尝试通过 Mongo Shell去执行索引的创建,发现返回了同样的错误。
    通过将数据库中的索引删除,或更正为 unique=true 之后可以解决当前的问题。

    从严谨度上看,一个索引冲突导致 SpringBoot 服务启动不了,是可以接受的。
    但从灵活性来看,是否有某些方式能禁用索引的自动创建,或者仅仅是打印日志呢?

    尝试 google spring data mongodb disable index creation
    发现 JIRA-DATAMONGO-1201在2015年就已经提出,至今未解决。

    stackoverflow 找到许多同样问题
    但大多数的解答是不采用索引注解,选择其他方式对索引进行管理。

    这些结果并不能令人满意。

    尝试查看 spring-data-mongo 的机制,定位到 MongoPersistentEntityIndexCreator类:

    1. 初始化方法中,会根据 MappingContext(实体映射上下文)中已有的实体去创建索引
    1 public MongoPersistentEntityIndexCreator(MongoMappingContext mappingContext, MongoDbFactory mongoDbFactory,
    2             IndexResolver indexResolver) {
    3         ...
    4         //根据已有实体创建
    5         for (MongoPersistentEntity<?> entity : mappingContext.getPersistentEntities()) {
    6             checkForIndexes(entity);
    7         }
    8     }

    2. 在接收到MappingContextEvent时,创建对应实体的索引

     1     public void onApplicationEvent(MappingContextEvent<?, ?> event) {
     2 
     3         if (!event.wasEmittedBy(mappingContext)) {
     4             return;
     5         }
     6 
     7         PersistentEntity<?, ?> entity = event.getPersistentEntity();
     8 
     9         // Double check type as Spring infrastructure does not consider nested generics
    10         if (entity instanceof MongoPersistentEntity) {
    11             //创建单个实体索引
    12             checkForIndexes((MongoPersistentEntity<?>) entity);
    13         }
    14     }

    MongoPersistentEntityIndexCreator是通过MongoTemplate引入的,如下:

     1     public MongoTemplate(MongoDbFactory mongoDbFactory, MongoConverter mongoConverter) {
     2 
     3         Assert.notNull(mongoDbFactory);
     4 
     5         this.mongoDbFactory = mongoDbFactory;
     6         this.exceptionTranslator = mongoDbFactory.getExceptionTranslator();
     7         this.mongoConverter = mongoConverter == null ? getDefaultMongoConverter(mongoDbFactory) : mongoConverter;
     8         ...
     9 
    10         // We always have a mapping context in the converter, whether it's a simple one or not
    11         mappingContext = this.mongoConverter.getMappingContext();
    12         // We create indexes based on mapping events
    13         if (null != mappingContext && mappingContext instanceof MongoMappingContext) {
    14             indexCreator = new MongoPersistentEntityIndexCreator((MongoMappingContext) mappingContext, mongoDbFactory);
    15             eventPublisher = new MongoMappingEventPublisher(indexCreator);
    16             if (mappingContext instanceof ApplicationEventPublisherAware) {
    17                 ((ApplicationEventPublisherAware) mappingContext).setApplicationEventPublisher(eventPublisher);
    18             }
    19         }
    20     }
    21 
    22 
    23     ...
    24     //MongoTemplate实现了 ApplicationContextAware,当ApplicationContext被实例化时被感知
    25     public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    26 
    27         prepareIndexCreator(applicationContext);
    28 
    29         eventPublisher = applicationContext;
    30         if (mappingContext instanceof ApplicationEventPublisherAware) {
    31             //MappingContext作为事件来源,向ApplicationContext发布
    32             ((ApplicationEventPublisherAware) mappingContext).setApplicationEventPublisher(eventPublisher);
    33         }
    34         resourceLoader = applicationContext;
    35     }
    36 
    37     ...
    38     //注入事件监听
    39     private void prepareIndexCreator(ApplicationContext context) {
    40 
    41         String[] indexCreators = context.getBeanNamesForType(MongoPersistentEntityIndexCreator.class);
    42 
    43         for (String creator : indexCreators) {
    44             MongoPersistentEntityIndexCreator creatorBean = context.getBean(creator, MongoPersistentEntityIndexCreator.class);
    45             if (creatorBean.isIndexCreatorFor(mappingContext)) {
    46                 return;
    47             }
    48         }
    49 
    50         if (context instanceof ConfigurableApplicationContext) {
    51             //使 IndexCreator 监听 ApplicationContext的事件
    52             ((ConfigurableApplicationContext) context).addApplicationListener(indexCreator);
    53         }
    54     }

    由此可见,MongoTemplate在初始化时,先通过MongoConverter 带入 MongoMappingContext,
    随后完成一系列初始化,整个过程如下:

    • 实例化 MongoTemplate;
    • 实例化 MongoConverter;
    • 实例化 MongoPersistentEntityIndexCreator;
    • 初始化索引(通过MappingContext已有实体);
    • Repository初始化 -> MappingContext 发布映射事件;
    • ApplicationContext 将事件通知到 IndexCreator;
    • IndexCreator 创建索引

    在实例化过程中,没有任何配置可以阻止索引的创建。

    四、解决问题

    从前面的分析中,可以发现问题关键在 IndexCreator,能否提供一个自定义的实现呢,答案是可以的!

    实现的要点如下

    1. 实现一个IndexCreator,可继承MongoPersistentEntityIndexCreator,去掉索引的创建功能;
    2. 实例化 MongoConverter和 MongoTemplate时,使用一个空的 MongoMappingContext对象避免初始化索引;
    3. 将自定义的IndexCreator作为Bean进行注册,这样在prepareIndexCreator方法执行时,
      原来的 MongoPersistentEntityIndexCreator不会监听ApplicationContext的事件
    4. IndexCreator 实现了ApplicationContext监听,接管 MappingEvent事件处理。

    实例化Bean

     1     @Bean
     2     public MongoMappingContext mappingContext() {
     3         return new MongoMappingContext();
     4     }
     5 
     6     // 使用 MappingContext 实例化 MongoTemplate
     7     @Bean
     8     public MongoTemplate mongoTemplate(MongoDbFactory mongoDbFactory, MongoMappingContext mappingContext) {
     9         MappingMongoConverter converter = new MappingMongoConverter(new DefaultDbRefResolver(mongoDbFactory),
    10                 mappingContext);
    11         converter.setTypeMapper(new DefaultMongoTypeMapper(null));
    12 
    13         MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, converter);
    14 
    15         return mongoTemplate;
    16     }

    自定义IndexCreator

     1     // 自定义IndexCreator实现
     2     @Component
     3     public static class CustomIndexCreator extends MongoPersistentEntityIndexCreator {
     4 
     5         // 构造器引用MappingContext
     6         public CustomIndexCreator(MongoMappingContext mappingContext, MongoDbFactory mongoDbFactory) {
     7             super(mappingContext, mongoDbFactory);
     8         }
     9 
    10         public void onApplicationEvent(MappingContextEvent<?, ?> event) {
    11             PersistentEntity<?, ?> entity = event.getPersistentEntity();
    12 
    13             // 获得Mongo实体类
    14             if (entity instanceof MongoPersistentEntity) {
    15                 System.out.println("Detected MongoEntity " + entity.getName());
    16                 
    17                 //可实现索引处理..
    18             }
    19         }
    20     }

    在这里 CustomIndexCreator继承了MongoPersistentEntityIndexCreator,将自动接管MappingContextEvent事件的监听。
    在业务实现上可以根据需要完成索引的处理!

    小结

    spring-data-mongo 提供了非常大的便利性,但在灵活性支持上仍然不足。上述的方法实际上有些隐晦,在官方文档中并未提及这样的方式。
    ORM-Mapping 框架在实现Schema映射处理时需要考虑校验级别,比如 Hibernate便提供了 none/create/update/validation 多种选择,毕竟这对开发者来说更加友好。
    期待 spring-data-mongo 在后续的演进中能尽快完善 Schema的管理功能!

    作者:美码师

    HDC.Cloud 华为开发者大会2020 即将于2020年2月11日-12日在深圳举办,是一线开发者学习实践鲲鹏通用计算、昇腾AI计算、数据库、区块链、云原生、5G等ICT开放能力的最佳舞台。

    欢迎报名参会

  • 相关阅读:
    对于大规模机器学习的理解和认识
    Failed to initialize NVML: GPU access blocked by the operating system
    ubuntu 当中添加开机启动服务
    洛谷P2882 [USACO07MAR]面对正确的方式Face The Right Way(贪心)
    注意注意!
    洛谷P5092 [USACO2004OPEN]Cube Stacking 方块游戏 (带权并查集)
    loj10017. 「一本通 1.2 练习 4」传送带(三分套三分)
    POJ1475 Pushing Boxes(BFS套BFS)
    CF451E Devu and Flowers(组合数)
    POJ2311 Cutting Game(博弈论)
  • 原文地址:https://www.cnblogs.com/huaweicloud/p/11867342.html
Copyright © 2020-2023  润新知