• apache kylin的一些注意事项(解决kylin报错Storage schema reading not supported)


    1.目前我所使用的kylin版本为2.6.2,有时在完成一次构建后会出现fail to locate kylin.properties的异常,如图所示

    经排查,定位到kylin源码中的

    org.apache.kylin.engine.mr.common.AbstractHadoopJob中的loadKylinPropsAndMetadata方法,如图所示


    我发现在执行cube构建任务时拆分的job会频繁调用这个方法,但这里这么写有些看不懂,不知道是个BUG还是有其他原因,总之这里不正确的更新了kylin_conf环境变量的路径。这个问题我会在后续升级版本中持续关注,目前的解决方法是,在kylin根目录下创建名为meta的文件夹,并将conf目录下的所有配置文件拷贝进去或者创建一个软连接。

    2.因为业务数据存储在hbase上,因此在使用kylin分析时,需要先将hbase的数据mapping到hive上,再在kylin中load hive table。但在kylin中load table报错,而且普通表load没问题,只有mapping hbase的表才会报错,报的异常为java.lang.UnsupportedOperationException: Storage schema reading not supported

    track输出为

    java.lang.RuntimeException: cannot get HiveTableMeta
    at org.apache.kylin.source.hive.HiveMetadataExplorer.loadTableMetadata(HiveMetadataExplorer.java:68)
    at org.apache.kylin.rest.service.TableService.extractHiveTableMeta(TableService.java:205)
    at org.apache.kylin.rest.service.TableService.loadHiveTablesToProject(TableService.java:131)
    at org.apache.kylin.rest.controller.TableController.loadHiveTables(TableController.java:113)
    at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: MetaException(message:java.lang.UnsupportedOperationException: Storage schema reading not supported)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6935)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5182)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields(HiveMetaStore.java:5144)
    at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
    at com.sun.proxy.$Proxy70.get_fields(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1845)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1839)
    at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78)
    at org.apache.kylin.source.hive.HiveMetadataExplorer.loadTableMetadata(HiveMetadataExplorer.java:66)
    ... 80 more
    报错原因是kylin在load表的时候,需要调用StorageSchemaReader接口的readSchema方法,默认的default是没有这个实现方法的就会报错。

    解决方法是配置hive-site.xml中的属性

    <property>
    <name>metastore.storage.schema.reader.impl</name>
    <value>org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader</value>
    </property>

    3.目前kylin的authorization(授权)部分采用kylin自带的权限机制,根据不同的角色来授予不同的权限。authentication(鉴权)部分采用ldap的账号体系来进行用户登陆认证(具体接入ldap的方法参见我的其他文章)。我的权限控制需求是,在ldap中控制具体的用户登陆名,密码等等,用户分为管理者和查询者且查询者只能查归属项目下指定的表(并非所有表都可查)。

    具体实现策略如下:

    (1)ldap: ou=kylin_user对应平台所有用户,uid为登录名,userPassword为密码(默认加密方式即可)。 ou=kylin_group对应平台角色,cn为角色名,member对应角色所分配的用户,目前所有角色分为admin角色和普通角色,即ou=kylin_group中的cn=admin表示admin角色,它对应的member用户即为admin用户,admin用户在kylin中具有所有操作的权限。其他未赋予角色的用户为普通角色。

    (2)在ldap中创建好了对应的用户和用户组之后,需要在kylin中配置ldap相关参数,具体参考我的kylin接入ldap篇。接入完成后,再在7070/kylin中使用admin账号登陆,点击左上角的“management project”按钮,选择对应的project,然后在access选项卡中 配置普通用户的角色,如图所示

    普通用户的角色有四种,分别是ADMIN,MANAGEMENT,OPERATOR,QUERY,admin是超管,默认不要赋给普通用户此权限。management是project的管理员,可以执行该project下的所有操作。operator是操作者,能执行该project下的model创建,cube创建构建,insight查询等等,但无法在management project中管理project。query是最低权限者,只能执行该project对应的insight查询,适用于普通的查询数据的用户。

    (3)普通查询用户需要再额外配置表权限,使特定用户只能查特定表(默认是query权限的用户可以查询该project下的所有表)。具体配置方法是使用高级用户,在首页model下的datasource选项卡中,在左侧选择需要配置权限的表,然后在右侧的access选项卡中 增加grant,选择对应的用户。或者删除某个用户使其无法查询该表。如图所示

  • 相关阅读:
    基于Python的人脸动漫转换
    let 与 var的区别
    【LeetCode】汇总
    【HDU】4632 Palindrome subsequence(回文子串的个数)
    【算法】均匀的生成圆内的随机点
    【LeetCode】725. Split Linked List in Parts
    【LeetCode】445. Add Two Numbers II
    【LeetCode】437. Path Sum III
    【LeetCode】222. Count Complete Tree Nodes
    【LeetCode】124. Binary Tree Maximum Path Sum
  • 原文地址:https://www.cnblogs.com/exmyth/p/15706495.html
Copyright © 2020-2023  润新知