• sqoop的安装


    Sqoop是一个用来完成Hadoop和关系型数据库中的数据相互转移的工具,

    他可以将关系型数据库(MySql,Oracle,Postgres等)中的数据导入Hadoop的HDFS中,

    也可以将HDFS的数据导入关系型数据库中。

    前期准备:

    1.Sqoop需要安装在成功部署的Hadoop平台上,并且要求Hadoop已经正常启动。hadoop集群的搭建

    2.安装了mysql数据库,并且已经启动.CentOS安装mysql源码包

    3.下载mysql-connector-java-5.1.12.tar.gz。

    安装过程;

    1.下载sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz压缩包。

    2.上传至/usr/local/src

    3.解压缩并且移动到上级目录(/usr/local)

    4.解压缩mysql-connector-java-5.1.12.tar.gz文件,

    并且复制文件mysql-connector-java-5.1.12-bin.jar到/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib中。

    5.配置环境变量  

    cd   /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/conf
    cp sqoop-env-template.sh sqoop-env.sh
    vim sqoop-env.sh

    根据情况添加配置信息 内容如下:

    # Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    # included in all the hadoop scripts with source command
    # should not be executable directly
    # also should not be passed any arguments, since we need original $*
    
    # Set Hadoop-specific environment variables here.
    
    #Set path to where bin/hadoop is available
    export HADOOP_COMMON_HOME=/usr/local/hadoop-2.6.5
    
    #Set path to where hadoop-*-core.jar is available
    export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.6.5
    
    #set the path to where bin/hbase is available
    export HBASE_HOME=/usr/local/hbase-1.4.8
    
    #Set the path to where bin/hive is available
    export HIVE_HOME=/usr/local/apache-hive-2.1.0-bin
    
    #Set the path for where zookeper config dir is
    export ZOOCFGDIR=/usr/local/zookeeper/conf

    6.配置linux的环境变量

    cd
    vim .bash_profile

    在结尾添加如下代码:

    #sqoop
    export SQOOP_HOME=/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/
    export PATH=$PATH:$SQOOP_HOME/bin

    7.保存退出;时文件生效:

    source .bash_profile

    8.测试与mysql的连接:

    cd   /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/bin
    sqoop list-databases --connect jdbc:mysql://localhost:3306 --username root -P

    输入密码后会显示具体的数据库列表

    9.查看具体数据库中的 表信息:

    sqoop list-tables --connect jdbc:mysql://localhost:3306/db1 --username root -P

    10.执行查询语句:

    sqoop -eval --connect jdbc:mysql://localhost:3306/db1 --username root  -P --query "select * from a limit 5"
  • 相关阅读:
    JavaScript最佳实践
    Ionic2学习笔记(3):Pipe
    广商14级软件工程分数:第十一回合
    Ionic2学习笔记(2):自定义Component
    Ionic2学习笔记(1):新建一个页面
    Ionic2学习笔记(0):HelloWorld
    采用Kettle分页处理大数据量抽取任务
    广商14级软件工程分数:第十回合
    广商14级软件工程分数:第九回合
    广商14级软件工程-项目介绍视频汇总
  • 原文地址:https://www.cnblogs.com/guoyansi19900907/p/9960562.html
Copyright © 2020-2023  润新知