• spark报错:Class org.apache.mapreduce.io.compress.GzipCodec not found


    现象:在进行spark本地IDEA运行的时候,报错

    报错:

    Exception in thread "main" java.lang.RuntimeException: Error in configuring object
         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
         at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:185)
         at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:198)
         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
         at scala.Option.getOrElse(Option.scala:121)
         at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
         at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:46)
         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
         at scala.Option.getOrElse(Option.scala:121)
         at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)
         at org.apache.spark.rdd.RDD.count(RDD.scala:1166)
         at spark._core.describe.sparkRunJob$.main(sparkRunJob.scala:20)
         at spark._core.describe.sparkRunJob.main(sparkRunJob.scala)
    Caused by: java.lang.reflect.InvocationTargetException
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
         ... 17 more
    Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.mapreduce.io.compress.GzipCodec not found.
         at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
         at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180)
         at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
         ... 22 more
    Caused by: java.lang.ClassNotFoundException: Class org.apache.mapreduce.io.compress.GzipCodec not found
         at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2171)
         at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
         ... 24 more


    原因: 程序在运行的时候回去读取配置文件,在IDEA中也就是配置在resource中的core.xml文件,xml文件中如果配置以下属性:

    <property>
    <name>io.compression.codecs</name>
    <value>org.apache.mapreduce.io.compress.GzipCodec,org.apache.mapreduce.io.compress.DefaultCodec,org.apache.mapreduce.io.compress.SnappyCodec</value>
    </property>

    ,程序如果发现没有这几个类对应的jar包的话,就会包上面的错,如果已经导入了上面几个类对应的包的话,在配置上面这个属性的时候,不要有换行符,不要有空格等,否则会引起另外的错误

    解决方法:

    第一种:本地测试可以将这个配置给注释掉

    第二种,如果是生产上的话,则需要将报错的类对应的jar包放到对应的额lib目录下

  • 相关阅读:
    Prime Land(poj 1365)
    备用交换机(cogs 8)
    救命(洛谷 U4525)
    消息传递(cogs 1001)
    SQLite 使用主键,ROWID 及自增列
    SqlHelper 类
    一个继承的 DataGridView
    给JAVA的eclipse IDE 在线安装 SVN插件 / 给 eclipse 添加打开所在的文件夹功能
    实时更新DataGridView 合计值
    导出为文本格式
  • 原文地址:https://www.cnblogs.com/Gxiaobai/p/12682455.html
Copyright © 2020-2023  润新知