zip transformation算子,将两个RDD中的元素(KV格式/非KV格式)变成一个KV格式的RDD,两个RDD的每个分区元素个数必须相同。
spark.sparkContext.setLogLevel("error")
spark.sparkContext.setLogLevel("error") val kzc=spark.sparkContext.parallelize(1.to(10),2) val bd=spark.sparkContext.parallelize(List("a","b","c","d","e","f","g","h","i","j"),2) kzc.zip(bd).collect().foreach(println(_))
zipPartitions
spark.sparkContext.setLogLevel("error") val kzc=spark.sparkContext.parallelize(1.to(10),2) val bd=spark.sparkContext.parallelize(List("a","b","c","d","e","f"),2) val res=kzc.zipPartitions(bd){ (iterator1,iterator2)=>{ val result=new scala.collection.mutable.ListBuffer[String]() while(iterator1.hasNext && iterator2.hasNext){ result.append(iterator1.next()+"|"+iterator2.next()) } result.iterator } } res.collect().foreach(println(_))
zipWithIndex 该函数将RDD中的元素和这个元素在RDD中的索引号(从0开始)组合成(K,V)对。
spark.sparkContext.setLogLevel("error") val kzc=spark.sparkContext.parallelize(1.to(10),2) val bd=spark.sparkContext.parallelize(List("a","b","c","d","e","f"),2) bd.zipWithIndex().collect().foreach(println(_))