• spark读取写入jdbc.,Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()


     df.write
              .option("truncate", "true")
              .option("driver", mysqlDriver)
              .mode(SaveMode.Overwrite).jdbc(url, table, pro)
    

     需要配置某个jdbc类型驱动driver,option("driver", mysqlDriver),不配置的话不会通过url前缀去寻找的,会抛出错误

    1/01/26 14:07:35 INFO LineBufferedStream: stdout: java.lang.InstantiationException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.db.RelationDb.dataFrameInsert(RelationDb.scala:56)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:99)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:26)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:39)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$.run(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp.run(RecommendAnalysisApp.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest$.main(BulkLoadTest.scala:65)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest.main(BulkLoadTest.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:180)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.security.AccessController.doPrivileged(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at javax.security.auth.Subject.doAs(Subject.java:422)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

  • 相关阅读:
    242
    241
    240
    「音乐」(G)I-DLE—화(火花) (HWAA)
    「音乐」Red Velvet—Psycho
    「音乐」Taylor Swift—the lakes (bonus track)
    「笔记」DP简单笔记
    「游记」CSP2020-S2游记
    「音乐」BLACKPINK-Ice Cream (with Selena Gomez)
    matlab 2021a linux 版本下载
  • 原文地址:https://www.cnblogs.com/hejunhong/p/14330160.html
Copyright © 2020-2023  润新知