• Spark获取DataFrame中列的几种姿势--col,$,column,apply


    1.doc上的解释(https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Column.html)
     df("columnName")            // On a specific DataFrame.
       col("columnName")           // A generic column no yet associated with a DataFrame.
       col("columnName.field")     // Extracting a struct field
       col("`a.column.with.dots`") // Escape `.` in column names.
       $"columnName"               // Scala short hand for a named column.
       expr("a + 1")               // A column that is constructed from a parsed SQL Expression.
       lit("abc")                  // A column that produces a literal (constant) value.

    2.使用时涉及到的的包

       import spark.implicits._
       import org.apache.spark.sql.functions._
       import org.apache.spark.sql.Column
    3.示例
    scala> val idCol = $"id"
    idCol: org.apache.spark.sql.ColumnName = id
     
    scala> val idCol = col("id")
    idCol: org.apache.spark.sql.Column = id
     
    scala> val idCol = column("id")
    idCol: org.apache.spark.sql.Column = id
    scala> val dataset = spark.range(5).toDF("text")
    dataset: org.apache.spark.sql.DataFrame = [text: bigint]
     
    scala> val textCol = dataset.col("text")
    textCol: org.apache.spark.sql.Column = text
     
    scala> val textCol = dataset.apply("text")
    textCol: org.apache.spark.sql.Column = text
     
    scala> val textCol = dataset("text")
    textCol: org.apache.spark.sql.Column = text
  • 相关阅读:
    Sql中使用With创建多张临时表
    sql(join on 和where的执行顺序)
    什么是正则化
    ETL讲解(转)
    MySQL等 SQL语句在线练习
    Sublime text 3 --html
    Sublime text 3 搭建Python3 IDE
    地区列车经过查询
    Lasso回归算法: 坐标轴下降法与最小角回归法小结
    完全卸载VMware
  • 原文地址:https://www.cnblogs.com/itboys/p/10962665.html
Copyright © 2020-2023  润新知