• ALS部署Spark集群入坑记


    [Stage 236:>                                                      (0 + 0) / 400]17/12/04 09:45:55 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError
    java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1841)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1534)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    scala.collection.immutable.$colon$colon.writeObject(List.scala:379)
    sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    

    栈溢出原因:由于ASL的迭代次数过多,DAG过深,导致RDD的lineage,栈溢出。
    现有解决:1.迭代次数减少。
    2.利用checkpoint?
    参考:

    1. 记一次spark mllib stackoverflow踩坑
    2. 一个 KCore 算法引发的 StackOverflow 奇案
    3. Spark出现java.lang.stackoverflowerror的解决方法
    4. Spark sql解析异常java.lang.StackOverflowError处理
    5. 由于长RDD谱系,Stackoverflow

    其他:

    1. spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable
  • 相关阅读:
    python 赋值操作的知识点
    python while循环语句
    python dict遍历
    python列表的切片操作
    Python做下载器需要掌握哪些
    BeautifulSoup已经安装,但仍提示No module named
    python 列表循环输出中文
    python 字符串split (string split)
    python 调用解释器
    分享python字符串去除空格的知识点
  • 原文地址:https://www.cnblogs.com/hexu105/p/7975922.html
Copyright © 2020-2023  润新知