• 原创:自定义spark GraphX中的collectNeighborIds方法


    /**
    * 自定义收集VertexId的neighborIds
    * @author TongXueQiang
    */
    def collectNeighborIds[T,U](edgeDirection:EdgeDirection,graph:Graph[T,U])(implicit m:scala.reflect.ClassTag[T],n:scala.reflect.ClassTag[U]):VertexRDD[Array[VertexId]] = {
    val nbrs = graph.mapReduceTriplets[Array[VertexId]](
    //map函数
    edgeTriplets => {
    val msgTosrc = (edgeTriplets.srcId,Array(edgeTriplets.dstId));
    val msgTodst = (edgeTriplets.dstId,Array(edgeTriplets.srcId));
    edgeDirection match {
    case EdgeDirection.Either =>Iterator(msgTosrc,msgTodst)
    case EdgeDirection.Out => Iterator(msgTosrc)
    case EdgeDirection.In => Iterator(msgTodst)
    case EdgeDirection.Both => throw new SparkException("It doesn't make sense to collect neighbors without a " + "direction.(EdgeDirection.Both is not supported.use EdgeDirection.Either instead.)")
    }
    },_ ++ _)//reduce函数
    nbrs
    }
    测试:
    object Test {
      
      System.setProperty("hadoop.home.dir","D://hadoop-2.6.2");
      val conf = new SparkConf().setMaster("local").setAppName("SparkGraph");
      val sc = new SparkContext(conf);

      def main(args:Array[String]):Unit = {
        val graph = GraphGenerators.logNormalGraph(sc,numVertices = 100).map((id,_) => id.toDouble);
        collectNeighborIds(EdgeDirection.In,graph).foreach(line => {print(line._1+":"); for (elem <- line._2) {print(elem + " ")};println;});

    }



    }
  • 相关阅读:
    selenium模块---操作浏览器
    mock模块学习---模拟接口返回数据
    fiddler配置和使用
    css 08-CSS属性:定位属性
    css 07-浮动
    css 06-CSS盒模型详解
    css 05-CSS样式表的继承性和层叠性
    css 04-CSS选择器:伪类
    css 03-CSS样式表和选择器
    css 02-CSS属性:背景属性
  • 原文地址:https://www.cnblogs.com/txq157/p/6001401.html
Copyright © 2020-2023  润新知