• ALS部署Spark集群入坑记


    [Stage 236:>                                                      (0 + 0) / 400]17/12/04 09:45:55 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError
    java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1841)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1534)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    scala.collection.immutable.$colon$colon.writeObject(List.scala:379)
    sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    

    栈溢出原因:由于ASL的迭代次数过多,DAG过深,导致RDD的lineage,栈溢出。
    现有解决:1.迭代次数减少。
    2.利用checkpoint?
    参考:

    1. 记一次spark mllib stackoverflow踩坑
    2. 一个 KCore 算法引发的 StackOverflow 奇案
    3. Spark出现java.lang.stackoverflowerror的解决方法
    4. Spark sql解析异常java.lang.StackOverflowError处理
    5. 由于长RDD谱系,Stackoverflow

    其他:

    1. spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable
  • 相关阅读:
    java使用递归删除非空目录
    关于Java日期的两道例题
    equals和==的区别
    从键盘读入个数不确定的整数,并判断读入的正数和负数的个数,输入为0时结束程序。
    输出所有的水仙花数
    99乘法表
    switch
    next()、nextInt()
    流程控制
    Scanner从键盘输入
  • 原文地址:https://www.cnblogs.com/hexu105/p/7975922.html
Copyright © 2020-2023  润新知