Parquet是面向分析型业务得列式存储格式
编程方式加载数据
代码示例
package wujiadong_sparkSQL
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
/**
* Created by Administrator on 2017/2/3.
*/
object ParquetLoadData {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("ParquetLoadData")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val usersDF = sqlContext.read.parquet("hdfs://master:9000/student/2016113012/spark/users.parquet")
usersDF.registerTempTable("t_users")
//查询name
val usersNameDF = sqlContext.sql("select name from t_users")
//转换成RDD并执行相关操作
usersNameDF.rdd.map(row => "Name:"+row(0)).collect().foreach(username => println(username))
}
}
运行结果
hadoop@master:~/wujiadong$ spark-submit --class wujiadong_sparkSQL.ParquetLoadData --executor-memory 500m --total-executor-cores 2 /home/hadoop/wujiadong/wujiadong.spark.jar
17/02/03 14:36:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/03 14:36:02 INFO Slf4jLogger: Slf4jLogger started
17/02/03 14:36:03 INFO Remoting: Starting remoting
17/02/03 14:36:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.131:40895]
17/02/03 14:36:07 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
17/02/03 14:36:20 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
17/02/03 14:36:21 INFO CodecPool: Got brand-new decompressor [.snappy]
Name:Alyssa
Name:Ben
17/02/03 14:36:21 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/02/03 14:36:21 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
自动分区
hadoop@master:~$ hadoop fs -mkdir /student/2016113012/spark/users
hadoop@master:~$ hadoop fs -mkdir /student/2016113012/spark/users/gender=male/
hadoop@master:~$ hadoop fs -mkdir /student/2016113012/spark/users/gender=male/country=us
hadoop@master:~/wujiadong$ hadoop fs -put users.parquet /student/2016113012/spark/users/gender=male/country=us
hadoop@master:~/wujiadong$ spark-submit --class wujiadong_sparkSQL.ParquetPartitionTest --executor-memory 500m --total-executor-cores 2 /home/hadoop/wujiadong/wujiadong.spark.jar
17/02/03 15:13:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/03 15:13:43 INFO Slf4jLogger: Slf4jLogger started
17/02/03 15:13:43 INFO Remoting: Starting remoting
17/02/03 15:13:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.131:37709]
17/02/03 15:13:46 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
17/02/03 15:13:59 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
17/02/03 15:13:59 INFO CodecPool: Got brand-new decompressor [.snappy]
+------+--------------+----------------+------+-------+
| name|favorite_color|favorite_numbers|gender|country|
+------+--------------+----------------+------+-------+
|Alyssa| null| [3, 9, 15, 20]| male| us|
| Ben| red| []| male| us|
+------+--------------+----------------+------+-------+
17/02/03 15:14:00 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/02/03 15:14:00 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
自动推断出了性别和国家
合并元数据
1）读取parquet文件时，将数据源的选项mergeSchema，设置为true
2）使用SQLContext.setConf()方法，将spark.sql.parquet.mergeSchema参数设置为true
案例：合并学生的基本信息和成绩的元数据