• Accessing data in Hadoop using dplyr and SQL


    If your primary objective is to query your data in Hadoop to browse, manipulate, and extract it into R, then you probably want to use SQL. You can write SQL code explicitly to interact with Hadoop, or you can write SQL code implicitly with dplyr. The dplyrpackage has a generalized backend for data sources that translates your R code into SQL. You can use RStudio and dplyr to work with several of the most popular software packages in the Hadoop ecosystem, including Hive, Impala, HBase and Spark.

    There are two methods for accessing data in Hadoop using dplyr and SQL.

    ODBC

    You can connect R and RStudio to Hadoop with an ODBC connection. This effectively treats Hadoop like any other data source (i.e., as if Hadoop were a relational database). You will need a data source specific driver (e.g., Hive, Impala, HBase) installed on your desktop or your sever. You will also need a few R packages. We recommend using these R packages: DBIdplyr, and odbc. Note that the dplyr package may also reference the dbplyr package to help translate R into specific variants of SQL. You can use the odbc package to create a connection with Hadoop and run queries:

    library(odbc)

    con <- dbConnect(odbc::odbc(), driver = <driver>, host = <host>, dbname = <dbname>, user = <user>, password = <password>, port = 10000)

    tbl(con, "mytable") # dplyr
    dbGetQuery(con, "SELECT * FROM mytable") # SQL

    dbDisconnect(con)

    Spark

    If you are running Spark on Hadoop, you may also elect to use the sparklyr package to access your data in HDFS. Spark is a general engine for large-scale data processing, and it supports SQL. The sparklyr package communicates with the Spark API to run SQL queries, and it also has a dplyr backend. You can use sparklyr to create a connect with Spark run queries:

    library(sparklyr)
    
    con <- spark_connect(master = "yarn-client")

    tbl(con, "mytable") # dplyr
    dbGetQuery(con, "SELECT * FROM mytable") # SQL

    spark_disconnect(con)


    转自:https://support.rstudio.com/hc/en-us/articles/115008241668-Accessing-data-in-Hadoop-using-dplyr-and-SQL
  • 相关阅读:
    SQL Server 2008登录错误:无法连接到(local)解决方法
    HTML5 学习
    DNS服务器的配置与管理
    如何把TOMCAT 添加到服务中自动启动
    如何获取WIN10 Program Files 文件夹下的文件操作权限
    Oracle PL/SQL入门语法点
    【Oracle XE系列之三】使用OMF方式手工创建Oracle XE数据库
    【Oracle XE系列之二】PLSQL Developer 远程连接Oracle XE数据库
    【Oracle XE系列之一】Windows10_X64环境 安装Oracle XE11gR2 X64数据库
    Spark 调优
  • 原文地址:https://www.cnblogs.com/payton/p/8758893.html
Copyright © 2020-2023  润新知