• Hadoop 2.7.3 完全分布式维护-简单测试篇


    1. 测试MapReduce Job

     1.1 上传文件到hdfs文件系统

    $ jps
    15520 Jps
    13426 SecondaryNameNode
    14003 JobHistoryServer
    13211 NameNode
    13612 ResourceManager
    $ jps > infile
    $ hadoop fs -mkdir /inputdir
    $ hadoop fs -put infile /inputdir
    $ hadoop fs -ls /inputdir
    Found 1 items
    -rw-r--r--   3 hduser supergroup         94 2017-09-01 11:02 /inputdir/infile

     1.2 进行word count计算

    $ hadoop jar /usr/local/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /inputdir /outputdir
    17/09/01 11:04:37 INFO client.RMProxy: Connecting to ResourceManager at /172.16.101.55:8032
    17/09/01 11:04:39 INFO input.FileInputFormat: Total input paths to process : 1
    17/09/01 11:04:39 INFO mapreduce.JobSubmitter: number of splits:1
    17/09/01 11:04:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504106569900_0001
    17/09/01 11:04:41 INFO impl.YarnClientImpl: Submitted application application_1504106569900_0001
    17/09/01 11:04:41 INFO mapreduce.Job: The url to track the job: http://sht-sgmhadoopnn-01:8088/proxy/application_1504106569900_0001/
    17/09/01 11:04:41 INFO mapreduce.Job: Running job: job_1504106569900_0001
    17/09/01 11:04:58 INFO mapreduce.Job: Job job_1504106569900_0001 running in uber mode : false
    17/09/01 11:04:58 INFO mapreduce.Job:  map 0% reduce 0%
    17/09/01 11:05:06 INFO mapreduce.Job:  map 100% reduce 0%
    17/09/01 11:05:15 INFO mapreduce.Job:  map 100% reduce 100%
    17/09/01 11:05:16 INFO mapreduce.Job: Job job_1504106569900_0001 completed successfully
    17/09/01 11:05:16 INFO mapreduce.Job: Counters: 49
        File System Counters
            FILE: Number of bytes read=160
            FILE: Number of bytes written=238465
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
            HDFS: Number of bytes read=200
            HDFS: Number of bytes written=114
            HDFS: Number of read operations=6
            HDFS: Number of large read operations=0
            HDFS: Number of write operations=2
        Job Counters 
            Launched map tasks=1
            Launched reduce tasks=1
            Data-local map tasks=1
            Total time spent by all maps in occupied slots (ms)=5960
            Total time spent by all reduces in occupied slots (ms)=6543
            Total time spent by all map tasks (ms)=5960
            Total time spent by all reduce tasks (ms)=6543
            Total vcore-milliseconds taken by all map tasks=5960
            Total vcore-milliseconds taken by all reduce tasks=6543
            Total megabyte-milliseconds taken by all map tasks=6103040
            Total megabyte-milliseconds taken by all reduce tasks=6700032
        Map-Reduce Framework
            Map input records=5
            Map output records=10
            Map output bytes=134
            Map output materialized bytes=160
            Input split bytes=106
            Combine input records=10
            Combine output records=10
            Reduce input groups=10
            Reduce shuffle bytes=160
            Reduce input records=10
            Reduce output records=10
            Spilled Records=20
            Shuffled Maps =1
            Failed Shuffles=0
            Merged Map outputs=1
            GC time elapsed (ms)=223
            CPU time spent (ms)=2280
            Physical memory (bytes) snapshot=426209280
            Virtual memory (bytes) snapshot=4179288064
            Total committed heap usage (bytes)=315097088
        Shuffle Errors
            BAD_ID=0
            CONNECTION=0
            IO_ERROR=0
            WRONG_LENGTH=0
            WRONG_MAP=0
            WRONG_REDUCE=0
        File Input Format Counters 
            Bytes Read=94
        File Output Format Counters 
            Bytes Written=114

     1.3 查看wordcount结果

    $ hadoop fs -ls /outputdir
    Found 2 items
    -rw-r--r--   3 hduser supergroup          0 2017-09-01 11:05 /outputdir/_SUCCESS
    -rw-r--r--   3 hduser supergroup        114 2017-09-01 11:05 /outputdir/part-r-00000
    $ hadoop fs -cat /outputdir/part-r-00000
    13211    1
    13426    1
    13612    1
    14003    1
    15541    1
    JobHistoryServer    1
    Jps    1
    NameNode    1
    ResourceManager    1
    SecondaryNameNode    1

    2. 测试hdfs分布式存储

    2.1 上传测试文件

    $ ls -lh hadoop-2.7.3.tar.gz 
    -rw-r--r-- 1 root root 205M May  5 09:01 hadoop-2.7.3.tar.gz
    $ hadoop fs -put hadoop-2.7.3.tar.gz /inputdir
    $ hadoop fs -ls -h /inputdir
    Found 2 items
    -rw-r--r--   3 hduser supergroup    204.2 M 2017-09-01 11:09 /inputdir/hadoop-2.7.3.tar.gz
    -rw-r--r--   3 hduser supergroup         94 2017-09-01 11:02 /inputdir/infile

    2.2 查看datanode副本信息

  • 相关阅读:
    Kubernetes日常维护命令
    4-docker的三要素
    3-docker的安装
    2-docker介绍
    1-为什么要使用docker
    块存储、文件存储、对象存储意义及差异
    ceph分布式存储的搭建
    YAML入门:以创建一个Kubernetes deployment为例
    Zabbix通过SQL语句从数据库获取数据说明
    图解HTTP--03--http报文内的信息
  • 原文地址:https://www.cnblogs.com/ilifeilong/p/7462283.html
Copyright © 2020-2023  润新知