• centos 部署 SparkR


    ---恢复内容开始---

    环境配置——

    • 操作系统:CentOS 6.5
    • JDK版本:1.7.0_67
    • Hadoop集群版本:CDH 5.3.0

    安装过程——

    1、(1)安装R

    yum install -y R

    (2)出现问题:

    yum 安装软件时,报错:No package XXX available.

    [root@localhost ~]# yum -y install redis
    Loaded plugins: fastestmirror
    No package redis available.
    Nothing to do
    (3)解决方案

    配置一台CentOS 7.x或RHEL 7.x的系统使用Fedora EPEL  repos和第三方软件包安装各种软件包。请注意,这些软件包没有正式获得任何的CentOS或Red Hat的支持,但提供了许多流行的包和应用程序。

    显示以上错误,有以下2种原因:

    1、dns问题;

    2、镜像连接错误;

    DNS问题

    首先测试下网络连接,如果ping不通,需要设置下DNS。

    [root@localhost ~]# ping www.baidu.com
    XXXXXXXXXXXXXXXXX
    [root@localhost ~]# echo "nameserver 8.8.8.8">>/etc/resolv.conf
    

    再次ping即可ping通了。

    镜像连接错误问题

    具体原因:使用yum搜索某些rpm包,找不到包是因为CentOS是RedHat企业版编译过来的,去掉了所有关于版权问题的东西。安装EPEL后可以很好的解决这个问题。EPEL(Extra Packages for Enterprise Linux )即企业版Linux的扩展包,提供了很多可共Centos使用的组件,安装完这个以后基本常用的rpm都可以找到。

    Method #1 » 从企业版Linux库配置包中安装扩展包。(推荐 )

    [root@localhost ~]# sudo yum install epel-release
    Loaded plugins: fastestmirror, langpacks
    Loading mirror speeds from cached hostfile
    Resolving Dependencies
    --> Running transaction check
    ---> Package epel-release.noarch 0:7-5 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================
     Package                Arch             Version         Repository        Size
    ================================================================================
    Installing:
     epel-release           noarch           7-5             extras            14 k
    
    Transaction Summary
    ================================================================================
    Install  1 Package
    
    Total download size: 14 k
    Installed size: 24 k
    Is this ok [y/d/N]: y
    Downloading packages:
    epel-release-7-5.noarch.rpm                                |  14 kB   00:01     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : epel-release-7-5.noarch                                      1/1 
      Verifying  : epel-release-7-5.noarch                                      1/1 
    
    Installed:
      epel-release.noarch 0:7-5                                                     
    
    Complete!

    (4)最后运行下列代码 即可完成安装
    yum install -y R

    2、安装curl-devel(非常重要!否则无法进行RCurl包的安装,进而无法安装devtools)

    yum install -y curl-devel



    3.在R shell中安装devtools
    install.packages("RCurl")
    install.packages("devtools")

    install.packages("devtools")




    出现错误:
    
    [plain] view plain copy
    ------------------------- ANTICONF ERROR ---------------------------  
    Configuration failed because libcurl was not found. Try installing:  
     * deb: libcurl4-openssl-dev (Debian, Ubuntu, etc)  
     * rpm: libcurl-devel (Fedora, CentOS, RHEL)  
     * csw: libcurl_dev (Solaris)  
    If libcurl is already installed, check that 'pkg-config' is in your  
    PATH and PKG_CONFIG_PATH contains a libcurl.pc file. If pkg-config  
    is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:  
    R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'  
    --------------------------------------------------------------------  
    ERROR: configuration failed for package ‘curl’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/curl’  
    * installing *source* package ‘whisker’ ...  
    ** package ‘whisker’ successfully unpacked and MD5 sums checked  
    ** R  
    ** inst  
    ** preparing package for lazy loading  
    ** help  
    *** installing help indices  
    ** building package indices  
    ** testing if installed package can be loaded  
    * DONE (whisker)  
    * installing *source* package ‘digest’ ...  
    ** package ‘digest’ successfully unpacked and MD5 sums checked  
    ** libs  
    
    [plain] view plain copy
    configure: error: in `/tmp/RtmpkQMD9t/R.INSTALL6c1166a0db0e/git2r':  
    configure: error:   
      ---------------------------------------------  
       OpenSSL library required  
      
       Please install:  
       libssl-dev (deb) or openssl-devel (rpm)  
      ---------------------------------------------  
    See `config.log' for more details  
    ERROR: configuration failed for package ‘git2r’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/git2r’  
    * installing *source* package ‘stringr’ ...  
    ** package ‘stringr’ successfully unpacked and MD5 sums checked  
    ** R  
    ** inst  
    ** preparing package for lazy loading  
    ** help  
    *** installing help indices  
    ** building package indices  
    ** installing vignettes  
    ** testing if installed package can be loaded  
    * DONE (stringr)  
    * installing *source* package ‘xml2’ ...  
    ** package ‘xml2’ successfully unpacked and MD5 sums checked  
    Package libxml-2.0 was not found in the pkg-config search path.  
    Perhaps you should add the directory containing `libxml-2.0.pc'  
    to the PKG_CONFIG_PATH environment variable  
    No package 'libxml-2.0' found  
    Package libxml-2.0 was not found in the pkg-config search path.  
    Perhaps you should add the directory containing `libxml-2.0.pc'  
    to the PKG_CONFIG_PATH environment variable  
    No package 'libxml-2.0' found  
    Using PKG_CFLAGS=  
    Using PKG_LIBS=-lxml2  
    ------------------------- ANTICONF ERROR ---------------------------  
    Configuration failed because libxml-2.0 was not found. Try installing:  
     * deb: libxml2-dev (Debian, Ubuntu, etc)  
     * rpm: libxml2-devel (Fedora, CentOS, RHEL)  
     * csw: libxml2_dev (Solaris)  
    If libxml-2.0 is already installed, check that 'pkg-config' is in your  
    PATH and PKG_CONFIG_PATH contains a libxml-2.0.pc file. If pkg-config  
    is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:  
    R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'  
    --------------------------------------------------------------------  
    ERROR: configuration failed for package ‘xml2’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/xml2’  
    * installing *source* package ‘memoise’ ...  
    ** package ‘memoise’ successfully unpacked and MD5 sums checked  
    ** R  
    ** preparing package for lazy loading  
    ** help  
    *** installing help indices  
    ** building package indices  
    ** testing if installed package can be loaded  
    * DONE (memoise)  
    ERROR: dependency ‘curl’ is not available for package ‘httr’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/httr’  
    * installing *source* package ‘evaluate’ ...  
    ** package ‘evaluate’ successfully unpacked and MD5 sums checked  
    ** R  
    ** preparing package for lazy loading  
    ** help  
    *** installing help indices  
    ** building package indices  
    ** testing if installed package can be loaded  
    * DONE (evaluate)  
    * installing *source* package ‘roxygen2’ ...  
    ** package ‘roxygen2’ successfully unpacked and MD5 sums checked  
    ** libs  
    g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o  
    g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c isComplete.cpp -o isComplete.o  
    g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c leadingSpaces.cpp -o leadingSpaces.o  
    g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c parser.cpp -o parser.o  
    g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c wrapString.cpp -o wrapString.o  
    g++ -shared -L/usr/local/lib64 -o roxygen2.so RcppExports.o isComplete.o leadingSpaces.o parser.o wrapString.o -L/usr/local/lib64/R/lib -lR  
    installing to /home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/roxygen2/libs  
    ** R  
    ** inst  
    ** preparing package for lazy loading  
    ** help  
    *** installing help indices  
    ** building package indices  
    ** installing vignettes  
    ** testing if installed package can be loaded  
    * DONE (roxygen2)  
    ERROR: dependencies ‘curl’, ‘xml2’ are not available for package ‘rversions’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/rversions’  
    ERROR: dependencies ‘httr’, ‘curl’, ‘rversions’, ‘git2r’ are not available for package ‘devtools’  
    * removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/devtools’  
      
    The downloaded source packages are in  
            ‘/tmp/Rtmp1A16li/downloaded_packages’  
    Warning messages:  
    1: In install.packages("devtools") :  
      installation of package ‘curl’ had non-zero exit status  
    2: In install.packages("devtools") :  
      installation of package ‘git2r’ had non-zero exit status  
    3: In install.packages("devtools") :  
      installation of package ‘xml2’ had non-zero exit status  
    4: In install.packages("devtools") :  
      installation of package ‘httr’ had non-zero exit status  
    5: In install.packages("devtools") :  
      installation of package ‘rversions’ had non-zero exit status  
    6: In install.packages("devtools") :  
      installation of package ‘devtools’ had non-zero exit status  
    
    根据错误安装,我这里安装了
    
    [plain] view plain copy
    [root@feng03 ~]# yum install libcurl-devel  
    [root@feng03 ~]# yum install openssl-devel  
    [root@feng03 ~]# yum install libxml2-devel  
    
    进入R shell
    >install.packages("git2r")
    >install.packages("xml2")
    >install.packages("rversions")
    

      

    步骤4: 安装好rJava及devtools,接下来安装SparkR


    library(devtools) install_github("amplab-extras/SparkR-pkg", subdir="pkg")

    出现错误
    Launching sbt from sbt/sbt-launch-0.13.6.jar
    Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.6.jar
    make: *** [target/scala-2.10/sparkr-assembly-0.1.jar] 错误 1
    ERROR: compilation failed for package ‘SparkR’
    * removing ‘/home/ndscbigdata/R/x86_64-pc-linux-gnu-library/3.2/SparkR’
    错误: Command failed (1)

    解决:

    进入spark安装包中的R目录,直接输入如下命令,直接就DONE了。
     
    cd进入目录

    /data1/bigdata/spark-1.6.1/R/lib/SparkR

     R -e "devtools::install('.')"










    ---恢复内容结束---

  • 相关阅读:
    ViewDragHelper: ViewDragHelper的使用
    HashMap存入和取出数据顺序不一致
    服务器raid卡、磁盘缓存的配置策略
    关于LVS+Nginx为什么会被同时使用的思考
    JDT Icons-eclipse图标大全
    EclipseHTML/JS/CSS/JS 代码自动提示
    eclipse maven设置本地仓库
    mysql 补零操作(zerofill)
    mysql datetime字段默认值 设置 当前时间
    Timed out as no activity
  • 原文地址:https://www.cnblogs.com/awishfullyway/p/6371157.html
Copyright © 2020-2023  润新知