• RDD源码分析


    RDD源码解析

    一、

    RDD.scala

    - Resilient Distributed Dataset (RDD) 
        弹性分布式数据集
    
        弹性: 体现在计算上面
    
    - the basic abstraction in Spark
    - Represents an immutable
        val
    
        RDDA == RDDB
    
    - partitioned collection of elements
    - that can be operated on in parallel 
    
    RDDA: (1,2,3,4,5,6,7,8,9)               operated +1。(对RDD执行加1的操作)
        hadoop000:Partition1: (1,2,3)        +1
        hadoop001:Partition2: (4,5,6)        +1
        hadoop002:Partition3: (7,8,9)        +1
    
    对RDD上的所有元素进行加1,他在hadoop000,hadoop001,hadoop002三台机器上同时进行
    对RDD进行操作,也就是对`RDD上的所有分区进行操作`
    
    abstract class RDD[T: ClassTag](
        @transient private var _sc: SparkContext,
        @transient private var deps: Seq[Dependency[_]]
      ) extends Serializable with Logging {}
    
    
    关键字: (从上面获得的信息)
    1) 抽象类: RDD必然是有之类实现的,我们使用时直接使用其之类即可
    2) Serializable(序列化)
    3) Logging(日志)
    4) T (泛型)
    5) SparkContext (入口点)
    6) @transient(注解,暂时不懂)
    

    二、JdbcRDD.scala

    class JdbcRDD[T: ClassTag](
        sc: SparkContext,
        getConnection: () => Connection,
        sql: String,
        lowerBound: Long,
        upperBound: Long,
        numPartitions: Int,
        mapRow: (ResultSet) => T = JdbcRDD.resultSetToObjectArray _)
      extends RDD[T](sc, Nil) with Logging {
    

    三、 RDD五大特性:

    Internally, each RDD is characterized by five main properties:  
            (1、2、3必选,4、5可选)  
        1) A list of partitions    (分区列表)
        2) A function for computing each split/partition   (用于计算每个 分片/分区 的函数)
        3) A list of dependencies on other RDDs   (其它的RDD依赖关系)
                RDDA => RDDB => RDDC ==> RDDD  
        4) Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)    (可选的,用于键值RDD的分区程序,(例如: 说明RDD时哈希分区))
        5) Optionally, a list of preferred locations to compute each split on (e.g. block locations foran HDFS file) (可选的,用于计算每个首选位置的分片列表(例如: 块位置为HDFS文件))
    
       preferred locations (一个RDD,对应多个partition,所有有 s )
    
       深入理解 RDD 与 关键字 之间的关系
       Resilient、Distributed、Dataste   (弹性、分布式、数据集)
    
        (木桶原理,性能由最短的那块板决定,由最慢的任务决定计算性能)
    
    

    四、RDD五大特性和RDD源码中 方法的 对应关系

      1) def compute(split: Partition, context: TaskContext): Iterator[T]
      
      2) protected def getPartitions: Array[Partition]
      
      3) protected def getDependencies: Seq[Dependency[_]] = deps
      
      4) protected def getPreferredLocations(split: Partition): Seq[String] = Nil
      
      5) @transient val partitioner: Option[Partitioner] = None
    
    
  • 相关阅读:
    BottomNavigationView3个项以上不显示文字的问题
    ViewPager中height=wrap_content无效,ScrollView里边用ListView显示不全解决办法
    Gson解析
    Parcelable Android SDK提供,基于内存,读写高于硬盘
    Serializable序列化 && 坑
    OpenStack中添加Floating IP
    Hadoop
    Redis
    ELK
    OpenStack
  • 原文地址:https://www.cnblogs.com/suixingc/p/rdd-yuan-ma-fen-xi.html
Copyright © 2020-2023  润新知