RDD源码解析
一、
RDD.scala
- Resilient Distributed Dataset (RDD)
弹性分布式数据集
弹性: 体现在计算上面
- the basic abstraction in Spark
- Represents an immutable
val
RDDA == RDDB
- partitioned collection of elements
- that can be operated on in parallel
RDDA: (1,2,3,4,5,6,7,8,9) operated +1。(对RDD执行加1的操作)
hadoop000:Partition1: (1,2,3) +1
hadoop001:Partition2: (4,5,6) +1
hadoop002:Partition3: (7,8,9) +1
对RDD上的所有元素进行加1,他在hadoop000,hadoop001,hadoop002三台机器上同时进行
对RDD进行操作,也就是对`RDD上的所有分区进行操作`
abstract class RDD[T: ClassTag](
@transient private var _sc: SparkContext,
@transient private var deps: Seq[Dependency[_]]
) extends Serializable with Logging {}
关键字: (从上面获得的信息)
1) 抽象类: RDD必然是有之类实现的,我们使用时直接使用其之类即可
2) Serializable(序列化)
3) Logging(日志)
4) T (泛型)
5) SparkContext (入口点)
6) @transient(注解,暂时不懂)
二、JdbcRDD.scala
class JdbcRDD[T: ClassTag](
sc: SparkContext,
getConnection: () => Connection,
sql: String,
lowerBound: Long,
upperBound: Long,
numPartitions: Int,
mapRow: (ResultSet) => T = JdbcRDD.resultSetToObjectArray _)
extends RDD[T](sc, Nil) with Logging {
三、 RDD五大特性:
Internally, each RDD is characterized by five main properties:
(1、2、3必选,4、5可选)
1) A list of partitions (分区列表)
2) A function for computing each split/partition (用于计算每个 分片/分区 的函数)
3) A list of dependencies on other RDDs (其它的RDD依赖关系)
RDDA => RDDB => RDDC ==> RDDD
4) Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned) (可选的,用于键值RDD的分区程序,(例如: 说明RDD时哈希分区))
5) Optionally, a list of preferred locations to compute each split on (e.g. block locations foran HDFS file) (可选的,用于计算每个首选位置的分片列表(例如: 块位置为HDFS文件))
preferred locations (一个RDD,对应多个partition,所有有 s )
深入理解 RDD 与 关键字 之间的关系
Resilient、Distributed、Dataste (弹性、分布式、数据集)
(木桶原理,性能由最短的那块板决定,由最慢的任务决定计算性能)
四、RDD五大特性和RDD源码中 方法的 对应关系
1) def compute(split: Partition, context: TaskContext): Iterator[T]
2) protected def getPartitions: Array[Partition]
3) protected def getDependencies: Seq[Dependency[_]] = deps
4) protected def getPreferredLocations(split: Partition): Seq[String] = Nil
5) @transient val partitioner: Option[Partitioner] = None