搭建maven开发环境测试Hadoop组件HDFS文件系统的一些命令

1.PC已经安装Eclipse Software，测试平台windows10及Centos6.8虚拟机

2.新建maven project

        

3.打开pom.xml,maven工程项目的pom文件加载以下内容

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.neusoft</groupId>
    <artifactId>bigdata001</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>bigdata001</name>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.6.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.6.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.6.0</version>
    </dependency>
        <dependency>
        <groupId>jdk.tools</groupId>
        <artifactId>jdk.tools</artifactId>
        <version>1.7</version>
        <scope>system</scope>
        <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper -->
    <dependency>
        <groupId>org.apache.zookeeper</groupId>
        <artifactId>zookeeper</artifactId>
        <version>3.4.6</version>
    </dependency>
    

    </dependencies>
    
    <build>
        <plugins>
            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>
            <configuration>
                <encoding>UTF-8</encoding>
                <source>1.7</source>
                <target>1.7</target>
                <showWarning>true</showWarning>
            </configuration>
            </plugin>
        </plugins>
    </build>
</project>

 4.如下图所示，下一步在src/main/java/下新建hdfs包，并新建java class：FileSystemTest.java

         

5.FileSystemTest.java代码内容如下：

package Hdfs;

import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class FileSystemTest {
    public static void main(String[] args) throws Exception {
        FileSystem fileSystem = FileSystem.newInstance(new URI("hdfs://neusoft-master:9000"), new Configuration());
        FileStatus[] listStatus = fileSystem.listStatus(new Path("/"));
        for (FileStatus fileStatus : listStatus) {
            System.out.println(fileStatus);
        }
    }
}

6.运行结果如下所示

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
FileStatus{path=hdfs://neusoft-master:9000/hbase; isDirectory=true; modification_time=1483712306703; access_time=0; owner=root; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
FileStatus{path=hdfs://neusoft-master:9000/tmp; isDirectory=true; modification_time=1483709831059; access_time=0; owner=root; group=supergroup; permission=rwx------; isSymlink=false}
FileStatus{path=hdfs://neusoft-master:9000/user; isDirectory=true; modification_time=1483709981792; access_time=0; owner=root; group=supergroup; permission=rwxr-xr-x; isSymlink=false}

      运行java类可以显示hadoop的HDFS文件系统下面各个目录。

7.在VM虚拟机搭建的Hadoop伪分布式环境进行测试。

    

8.总结

    上述步骤总结了如何通过在Windows平台的Eclipse平台下通过编码实现查看，hdfs文件系统内容。

 备注：编程时：用到了hdfs://neusoft-master:9000的neusoft-master需要在windows及linux平台进行设置

    windows系统需要在C:WindowsSystem32driversetc以管理员打开并添加“192.168.191.130 NEUSOFT-MASTER”

    Linux系统需要修改主机名(vi /etc/sysconfig/network修改hostname)及修改vi /etc/hosts对应的ip和host