继上一篇文章介绍如何使用Pig处理HDFS上的数据，本文将介绍使用Apache Hive进行数据查询和处理。
Apache Hive简介
- 首先Hive是一款数据仓库软件
- 使用HiveQL来结构化和查询存放的数据
- 执行环境：MapReduce, Tez, Spark
- 数据存放：HDFS, HBase
- 使用场景：数据挖掘和分析，机器学习，即席查询等
Hive使用示例
- 还是使用passwd作为操作文件
beeline> !quit
[cloudera@quickstart ~]$ hdfs dfs -put /etc/passwd /tmp/
[cloudera@quickstart ~]$ hdfs dfs -ls /tmp/
Found 5 items
drwxrwxrwt - mapred mapred 0 2016-12-29 01:05 /tmp/hadoop-yarn
drwx-wx-wx - hive supergroup 0 2016-08-27 10:19 /tmp/hive
drwxrwxrwt - mapred hadoop 0 2016-08-10 14:37 /tmp/logs
-rw-r--r-- 1 cloudera supergroup 2559 2017-02-22 05:34 /tmp/passwd
- 使用beeline连接Hive
[cloudera@quickstart ~]$ beeline -u jdbc:hive2://
scan complete in 24ms
Connecting to jdbc:hive2://
Connected to: Apache Hive (version 1.1.0-cdh5.8.0)
Driver: Hive JDBC (version 1.1.0-cdh5.8.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.8.0 by Apache Hive
0: jdbc:hive2://>
- 建表并且插入数据
0: jdbc:hive2://> CREATE TABLE userinfo ( uname STRING, pswd STRING, uid INT, gid INT, fullname STRING, hdir STRING, shell STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ':' STORED AS TEXTFILE;
0: jdbc:hive2://> LOAD DATA INPATH '/tmp/passwd' OVERWRITE INTO TABLE userinfo;
0: jdbc:hive2://> select uname,fullname,hdir from userinfo order by unmame;
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 27.83 sec HDFS Read: 8767 HDFS Write: 1454 SUCCESS
Total MapReduce CPU Time Spent: 27 seconds 830 msec
OK
+----------------+-------------------------------+-------------------------------+--+
| uname | fullname | hdir |
+----------------+-------------------------------+-------------------------------+--+
| abrt | | /etc/abrt |
| adm | adm | /var/adm |
| apache | Apache | /var/www |
| avahi-autoipd | Avahi IPv4LL Stack | /var/lib/avahi-autoipd |
| bin | bin | /bin |
| cloudera | | /home/cloudera |
| cloudera-scm | Cloudera Manager | /var/lib/cloudera-scm-server |
...
总结
- 使用beeline进行对Hive交互访问，类似于sqlplus之于Oracle数据库
- 其它的交互工作好包括：Hive CLI, Hcatalog, WebHcat
- 相应的DDL, DML语法可以参考官方WIKI