• 测试Hadoop2.7.1


    三台机器 CentOS7(机器名分别为master-CentOS7、slave1-CentOS7、slave2-CentOS7),每台机器内存2G(迫于无奈,刚换了内存条)

    之前写了一篇“CentOS 7 安装Hadoop 2.7.1http://blog.csdn.net/noob_f/article/details/52356779

    wordcount统计单词

    master-CentOS7(已启动Hadoop集群)

    [root@master ~]# cd /usr/local/hadoop/
     [root@master hadoop]# bin/hdfs dfs -mkdir /test001
    [root@master hadoop]# bin/hdfs dfs -ls /
    Found 3 items
    drwxr-xr-x   - root supergroup          0 2016-09-01 19:41 /test001
    drwx------   - root supergroup          0 2016-08-29 20:26 /tmp
    drwxr-xr-x   - root supergroup          0 2016-08-29 20:26 /user
    [root@master hadoop]# ls
    bin  etc      lib      LICENSE.txt  NOTICE.txt  sbin   tmp
    dfs  include  libexec  logs         README.txt  share
    [root@master hadoop]# wc -l LICENSE.txt
    289 LICENSE.txt
    [root@master hadoop]# du -sh !$
    du -sh LICENSE.txt
    16K     LICENSE.txt
    [root@master hadoop]# bin/hdfs dfs -copyFromLocal ./LICENSE.txt /test001
    [root@master hadoop]# bin/hdfs dfs -ls /test001
    Found 1 items
    -rw-r--r--   2 root supergroup      15429 2016-09-01 19:46 /test001/LICENSE.txt
    [root@master hadoop]# bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /test001/LICENSE.txt /test001/
    [root@master hadoop]# echo $?
    255
    

    发现命令执行出错

    org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://192.168.1.182:9000/test001 already exists

    命令改一下

    [root@master hadoop]# bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /test001/LICENSE.txt /test001/wordcount
    16/09/02 17:09:35 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.182:8032
    16/09/02 17:09:36 INFO input.FileInputFormat: Total input paths to process : 1
    16/09/02 17:09:36 INFO mapreduce.JobSubmitter: number of splits:1
    16/09/02 17:09:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1472804584592_0003
    16/09/02 17:09:37 INFO impl.YarnClientImpl: Submitted application application_1472804584592_0003
    16/09/02 17:09:37 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1472804584592_0003/
    16/09/02 17:09:37 INFO mapreduce.Job: Running job: job_1472804584592_0003
    16/09/02 17:09:46 INFO mapreduce.Job: Job job_1472804584592_0003 running in uber mode : false
    16/09/02 17:09:46 INFO mapreduce.Job:  map 0% reduce 0%
    16/09/02 17:09:55 INFO mapreduce.Job:  map 100% reduce 0%
    16/09/02 17:10:04 INFO mapreduce.Job:  map 100% reduce 100%
    16/09/02 17:10:05 INFO mapreduce.Job: Job job_1472804584592_0003 completed successfully
    16/09/02 17:10:05 INFO mapreduce.Job: Counters: 49
            File System Counters
                    FILE: Number of bytes read=10992
                    FILE: Number of bytes written=252973
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=15539
                    HDFS: Number of bytes written=8006
                    HDFS: Number of read operations=6
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=2
            Job Counters
                    Launched map tasks=1
                    Launched reduce tasks=1
                    Data-local map tasks=1
                    Total time spent by all maps in occupied slots (ms)=6493
                    Total time spent by all reduces in occupied slots (ms)=6714
                    Total time spent by all map tasks (ms)=6493
                    Total time spent by all reduce tasks (ms)=6714
                    Total vcore-seconds taken by all map tasks=6493
                    Total vcore-seconds taken by all reduce tasks=6714
                    Total megabyte-seconds taken by all map tasks=6648832
                    Total megabyte-seconds taken by all reduce tasks=6875136
            Map-Reduce Framework
                    Map input records=289
                    Map output records=2157
                    Map output bytes=22735
                    Map output materialized bytes=10992
                    Input split bytes=110
                    Combine input records=2157
                    Combine output records=755
                    Reduce input groups=755
                    Reduce shuffle bytes=10992
                    Reduce input records=755
                    Reduce output records=755
                    Spilled Records=1510
                    Shuffled Maps =1
                    Failed Shuffles=0
                    Merged Map outputs=1
                    GC time elapsed (ms)=146
                    CPU time spent (ms)=2360
                    Physical memory (bytes) snapshot=312647680
                    Virtual memory (bytes) snapshot=1717682176
                    Total committed heap usage (bytes)=163123200
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters
                    Bytes Read=15429
            File Output Format Counters
                    Bytes Written=8006
    [root@master hadoop]# echo $?
    0
    [root@master hadoop]# bin/hdfs dfs -ls /test001/
    Found 2 items
    -rw-r--r--   2 root supergroup      15429 2016-09-01 19:46 /test001/LICENSE.txt
    drwxr-xr-x   - root supergroup          0 2016-09-02 17:10 /test001/wordcount
    [root@master hadoop]# bin/hdfs dfs -ls /test001/wordcount
    Found 2 items
    -rw-r--r--   2 root supergroup          0 2016-09-02 17:10 /test001/wordcount/_SUCCESS
    -rw-r--r--   2 root supergroup       8006 2016-09-02 17:10 /test001/wordcount/part-r-00000
    [root@master hadoop]# bin/hdfs dfs -cat /test001/wordcount/part-r-00000
    "AS     4
    "Contribution"  1
    "Contributor"   1
    "Derivative     1
    "Legal  1
    "License"       1
    "License");     1
    "Licensor"      1
    "NOTICE"        1
    "Not    1
    "Object"        1
    "Source"        1
    "Work"  1
    "You"   1
    "Your") 1
    "[]"    1
    "control"       1
    "printed        1
    "submitted"     1
    (50%)   1
    (C)     1
    (Don't  1
    (INCLUDING      2
    (INCLUDING,     2
    (a)     1
    (an     1
    (and    1
    (b)     1
    (c)     2
    (d)     1
    (except 1
    (http://www.one-lab.org)        1
    (http://www.opensource.org/licenses/bsd-license.php)    1
    (i)     1
    (ii)    1
    (iii)   1
    (including      3
    (or     3
    (such   1
    (the    1
    *       34
    */      3
    -       7
    /*      1
    /**     2
    034819  1
    1       1
    1.      1
    2-Clause        1
    2.      1
    2.0     1
    2.0,    1
    2004    1
    2005,   1
    2008,2009,2010  1
    2011-2014,      1
    3.      1
    4.      1
    5.      1
    6.      1
    7.      1
    8.      1
    9       1
    9.      1
    :       3
    A       3
    ADVISED 2
    AND     11
    ANY     10
    APACHE  1
    APPENDIX:       1
    ARE     2
    ARISING 2
    Accepting       1
    Additional      1
    All     2
    Apache  5
    Appendix        1
    BASIS,  2
    BE      2
    BSD     1
    BSD-style       1
    BUSINESS        2
    BUT     4
    BY      2
    CAUSED  2
    CONDITIONS      4
    CONSEQUENTIAL   2
    CONTRACT,       2
    CONTRIBUTORS    4
    COPYRIGHT       4
    CRC     1
    Catholique      1
    Collet. 1
    Commission      1
    Contribution    3
    Contribution(s) 3
    Contribution."  1
    Contributions)  1
    Contributions.  2
    Contributor     8
    Contributor,    1
    Copyright       5
    DAMAGE. 2
    DAMAGES 2
    DATA,   2
    DIRECT, 2
    DISCLAIMED.     2
    DISTRIBUTION    1
    Definitions.    1
    Derivative      17
    Disclaimer      1
    END     1
    EVEN    2
    EVENT   2
    EXEMPLARY,      2
    EXPRESS 2
    Entity  3
    Entity" 1
    European        1
    FITNESS 3
    FOR     6
    Fast    1
    File    1
    For     6
    GOODS   2
    Grant   2
    HADOOP  1
    HOLDERS 2
    HOWEVER 2
    Hadoop  1
    Header  1
    How     1
    However,        1
    IF      2
    IMPLIED 4
    IN      6
    INCIDENTAL,     2
    INCLUDING,      2
    INDIRECT,       2
    INTERRUPTION)   2
    IS      2
    IS"     4
    If      2
    In      1
    Institute       1
    January 1
    KIND,   2
    LIABILITY,      4
    LIABLE  2
    LICENSE 1
    LIMITED 4
    LOSS    2
    LZ      1
    LZ4     3
    Legal   3
    Liability.      2
    License 10
    License,        6
    License.        11
    License;        1
    Licensed        1
    Licensor        8
    Licensor,       1
    Limitation      1
    Louvain 1
    MERCHANTABILITY 2
    MERCHANTABILITY,        1
    Massachusetts   1
    NEGLIGENCE      2
    NO      2
    NON-INFRINGEMENT,       1
    NOT     4
    NOTICE  5
    Neither 1
    Notwithstanding 1
    OF      19
    ON      2
    OR      18
    OTHERWISE)      2
    OUT     2
    OWNER   2
    Object  4
    OneLab  1
    PARTICULAR      3
    POSSIBILITY     2
    PROCUREMENT     2
    PROFITS;        2
    PROVIDED        2
    PURPOSE 2
    PURPOSE.        1
    Patent  1
    REPRODUCTION,   1
    Redistribution  2
    Redistribution. 1
    Redistributions 4
    SERVICES;       2
    SHALL   2
    SOFTWARE        2
    SOFTWARE,       2
    SPECIAL,        2
    STRICT  2
    SUBCOMPONENTS:  1
    SUBSTITUTE      2
    SUCH    2
    Sections        1
    See     1
    Source  8
    Subject 2
    Submission      1
    TERMS   2
    THE     10
    THEORY  2
    THIS    4
    TITLE,  1
    TO,     4
    TORT    2
    Technology.     1
    The     3
    This    1
    To      1
    Trademarks.     1
    UCL     1
    USE     2
    USE,    3
    University      1
    Unless  3
    Use     1
    Version 2
    WARRANTIES      4
    WARRANTIES,     2
    WAY     2
    WHETHER 2
    WITHOUT 2
    Warranty        1
    Warranty.       1
    We      1
    While   1
    Work    20
    Work,   4
    Work.   1
    Works   12
    Works"  1
    Works,  2
    Works;  3
    Yann    1
    You     24
    Your    9
    [name   1
    [yyyy]  1
    a       21
    above   4
    above,  1
    acceptance      1
    accepting       2
    act     1
    acting  1
    acts)   1
    add     2
    addendum        1
    additional      4
    additions       1
    advised 1
    against 1
    against,        1
    agree   1
    agreed  3
    agreement       1
    algorithm       1
    all     3
    alleging        1
    alone   1
    along   1
    alongside       1
    also    1
    an      6
    and     51
    and/or  3
    annotations,    1
    any     28
    appear. 1
    applicable      3
    applies 1
    apply   2
    appropriate     1
    appropriateness 1
    archives.       1
    are     10
    arising 1
    as      15
    asserted        1
    associated      1
    assume  1
    at      3
    attach  1
    attached        1
    attribution     4
    author  1
    authorized      2
    authorship,     2
    authorship.     1
    available       1
    based   1
    be      7
    been    2
    behalf  5
    below). 1
    beneficial      1
    binary  4
    bind    1
    boilerplate     1
    brackets        1
    brackets!)      1
    but     5
    by      21
    by,     3
    calculation     1
    can     2
    cannot  1
    carry   1
    cause   2
    changed 1
    character       1
    charge  1
    choose  1
    claims  2
    class   1
    classes:        1
    code    5
    code,   2
    combination     1
    comment 1
    commercial      1
    common  1
    communication   3
    compiled        1
    compliance      1
    complies        1
    compression     1
    computer        1
    conditions      14
    conditions.     1
    conditions:     1
    configuration   1
    consequential   1
    consistent      1
    conspicuously   1
    constitutes     1
    construed       1
    contact 1
    contained       1
    contains        1
    content 1
    contents        1
    contract        2
    contract,       1
    contributors    1
    contributory    1
    control 2
    control,        1
    controlled      1
    conversions     1
    copies  1
    copy    3
    copyright       15
    copyright,      1
    counterclaim    1
    cross-claim     1
    customary       1
    damages 3
    damages,        1
    damages.        1
    date    1
    de      1
    defend, 1
    defined 1
    definition,     2
    deliberate      1
    derived 2
    describing      1
    description     1
    designated      1
    determining     1
    different       1
    direct  2
    direct, 1
    direction       1
    disclaimer      2
    disclaimer.     2
    discussing      1
    display 1
    display,        1
    distribute      3
    distribute,     2
    distributed     3
    distribution    3
    distribution,   1
    distribution.   2
    do      3
    document.       1
    documentation   3
    documentation,  2
    does    1
    each    4
    easier  1
    editorial       1
    either  2
    elaborations,   1
    electronic      1
    electronic,     1
    enclosed        2
    endorse 1
    entities        1
    entity  3
    entity, 1
    entity. 2
    even    1
    event   1
    example 1
    except  2
    excluding       3
    executed        1
    exercise        1
    exercising      1
    explicitly      1
    express 2
    failure 1
    fee     1
    fields  1
    fifty   1
    file    6
    file,   1
    file.   2
    filed.  1
    files   1
    files.  1
    files;  1
    following       10
    for     19
    for,    1
    form    10
    form,   4
    form.   1
    format. 1
    forms,  2
    forum   1
    found   1
    from    4
    from)   1
    from,   1
    generated       2
    give    1
    goodwill,       1
    governed        1
    governing       1
    grant   1
    granted 2
    granting        1
    grants  2
    grossly 1
    harmless        1
    has     2
    have    2
    hereby  2
    herein  1
    hold    1
    http://code.google.com/p/lz4/   1
    http://www.apache.org/licenses/ 1
    http://www.apache.org/licenses/LICENSE-2.0      1
    https://groups.google.com/forum/#!forum/lz4c    1
    identification  1
    identifying     1
    if      4
    implementation  1
    implied,        1
    implied.        1
    import, 1
    improving       1
    in      31
    inability       1
    incidental,     1
    include 3
    included        2
    includes        1
    including       5
    including,      1
    inclusion       2
    incorporated    2
    incurred        1
    indemnify,      1
    indemnity,      1
    indicated       1
    indirect,       2
    individual      3
    information.    1
    informational   1
    infringed       1
    infringement,   1
    institute       1
    intentionally   2
    interfaces      1
    irrevocable     2
    is      10
    issue   1
    its     4
    language        1
    law     3
    lawsuit)        1
    least   1
    legal   1
    liability       2
    liability.      1
    liable  1
    licensable      1
    license 7
    licenses        1
    licenses.       1
    limitation,     1
    limitations     1
    limited 4
    link    1
    list    4
    lists,  1
    litigation      2
    loss    1
    losses),        1
    made    1
    made,   1
    mailing 1
    make,   1
    making  1
    malfunction,    1
    managed 1
    management      1
    marked  1
    marks,  1
    materials       2
    may     10
    mean    10
    means   2
    mechanical      1
    media   1
    medium, 1
    meet    1
    merely  1
    met:    2
    modification,   2
    modifications   3
    modifications,  3
    modified        1
    modify  2
    modifying       1
    more    1
    must    8
    name    2
    name)   1
    names   2
    names,  1
    native  1
    necessarily     1
    negligence),    1
    negligent       1
    no      2
    no-charge,      2
    non-exclusive,  2
    nor     1
    normally        1
    not     11
    nothing 1
    notice  2
    notice, 5
    notices 9
    object  1
    obligations     1
    obligations,    1
    obtain  1
    of      75
    of,     3
    offer   1
    offer,  1
    on      11
    one     1
    only    4
    or      65
    or,     1
    org.apache.hadoop.util.bloom.*  1
    origin  1
    original        2
    other   9
    otherwise       3
    otherwise,      3
    out     1
    outstanding     1
    own     4
    owner   4
    owner.  1
    owner]  1
    ownership       2
    page"   1
    part    4
    patent  5
    patent, 1
    percent 1
    perform,        1
    permission      1
    permission.     1
    permissions     3
    permitted       2
    perpetual,      2
    pertain 2
    places: 1
    portions        1
    possibility     1
    power,  1
    preferred       1
    prepare 1
    prior   1
    product 1
    products        1
    project 2
    prominent       1
    promote 1
    provide 1
    provided        9
    provides        2
    public  1
    publicly        2
    purpose 2
    purposes        4
    readable        1
    reason  1
    reasonable      1
    received        1
    recipients      1
    recommend       1
    redistributing  2
    regarding       1
    remain  1
    replaced        1
    repository      1
    represent,      1
    representatives,        1
    reproduce       3
    reproduce,      1
    reproducing     1
    reproduction,   3
    required        4
    reserved.       2
    responsibility, 1
    responsible     1
    result  1
    resulting       1
    retain  2
    retain, 1
    revisions,      1
    rights  3
    risks   1
    royalty-free,   2
    same    1
    section)        1
    sell,   2
    sent    1
    separable       1
    separate        2
    service 1
    shall   15
    shares, 1
    should  1
    slicing-by-8    1
    software        3
    sole    1
    solely  1
    source  9
    source, 1
    special,        1
    specific        2
    src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c},    1
    src/main/native/src/org/apache/hadoop/util:     1
    state   1
    stated  2
    statement       1
    stating 1
    stoppage,       1
    subcomponents   2
    subject 1
    sublicense,     1
    submit  1
    submitted       2
    submitted.      1
    subsequently    1
    such    17
    supersede       1
    support,        1
    syntax  1
    systems 1
    systems,        1
    terminate       1
    terms   8
    terms.  1
    text    4
    that    25
    the     122
    their   2
    then    2
    theory, 1
    thereof 1
    thereof,        2
    thereof.        1
    these   1
    third-party     2
    this    22
    those   3
    through 1
    to      41
    tort    1
    tracking        1
    trade   1
    trademark,      1
    trademarks,     1
    transfer        1
    transformation  1
    translation     1
    types.  1
    under   10
    union   1
    unless  1
    use     8
    use,    4
    used    1
    using   1
    verbal, 1
    version 1
    warranties      1
    warranty        1
    warranty,       1
    was     1
    where   1
    wherever        1
    whether 4
    which   2
    whole,  2
    whom    1
    with    16
    within  8
    without 6
    work    5
    work,   2
    work.   1
    works   1
    worldwide,      2
    writing 1
    writing,        3
    written 2
    you     2
    your    4
    
    
    

    运行 PI 实例

    master-CentOS7(已启动Hadoop集群)

    [root@master ~]# cd /usr/local/hadoop/
    [root@master hadoop]# bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 100 100
    Number of Maps  = 100
    Samples per Map = 100
    Wrote input for Map #0
    Wrote input for Map #1
    Wrote input for Map #2
    Wrote input for Map #3
    Wrote input for Map #4
    Wrote input for Map #5
    Wrote input for Map #6
    Wrote input for Map #7
    Wrote input for Map #8
    Wrote input for Map #9
    Wrote input for Map #10
    Wrote input for Map #11
    Wrote input for Map #12
    Wrote input for Map #13
    Wrote input for Map #14
    Wrote input for Map #15
    Wrote input for Map #16
    Wrote input for Map #17
    Wrote input for Map #18
    Wrote input for Map #19
    Wrote input for Map #20
    Wrote input for Map #21
    Wrote input for Map #22
    Wrote input for Map #23
    Wrote input for Map #24
    Wrote input for Map #25
    Wrote input for Map #26
    Wrote input for Map #27
    Wrote input for Map #28
    Wrote input for Map #29
    Wrote input for Map #30
    Wrote input for Map #31
    Wrote input for Map #32
    Wrote input for Map #33
    Wrote input for Map #34
    Wrote input for Map #35
    Wrote input for Map #36
    Wrote input for Map #37
    Wrote input for Map #38
    Wrote input for Map #39
    Wrote input for Map #40
    Wrote input for Map #41
    Wrote input for Map #42
    Wrote input for Map #43
    Wrote input for Map #44
    Wrote input for Map #45
    Wrote input for Map #46
    Wrote input for Map #47
    Wrote input for Map #48
    Wrote input for Map #49
    Wrote input for Map #50
    Wrote input for Map #51
    Wrote input for Map #52
    Wrote input for Map #53
    Wrote input for Map #54
    Wrote input for Map #55
    Wrote input for Map #56
    Wrote input for Map #57
    Wrote input for Map #58
    Wrote input for Map #59
    Wrote input for Map #60
    Wrote input for Map #61
    Wrote input for Map #62
    Wrote input for Map #63
    Wrote input for Map #64
    Wrote input for Map #65
    Wrote input for Map #66
    Wrote input for Map #67
    Wrote input for Map #68
    Wrote input for Map #69
    Wrote input for Map #70
    Wrote input for Map #71
    Wrote input for Map #72
    Wrote input for Map #73
    Wrote input for Map #74
    Wrote input for Map #75
    Wrote input for Map #76
    Wrote input for Map #77
    Wrote input for Map #78
    Wrote input for Map #79
    Wrote input for Map #80
    Wrote input for Map #81
    Wrote input for Map #82
    Wrote input for Map #83
    Wrote input for Map #84
    Wrote input for Map #85
    Wrote input for Map #86
    Wrote input for Map #87
    Wrote input for Map #88
    Wrote input for Map #89
    Wrote input for Map #90
    Wrote input for Map #91
    Wrote input for Map #92
    Wrote input for Map #93
    Wrote input for Map #94
    Wrote input for Map #95
    Wrote input for Map #96
    Wrote input for Map #97
    Wrote input for Map #98
    Wrote input for Map #99
    Starting Job
    16/09/02 16:40:43 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.182:8032
    16/09/02 16:40:44 INFO input.FileInputFormat: Total input paths to process : 100
    16/09/02 16:40:45 INFO mapreduce.JobSubmitter: number of splits:100
    16/09/02 16:40:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1472804584592_0002
    16/09/02 16:40:45 INFO impl.YarnClientImpl: Submitted application application_1472804584592_0002
    16/09/02 16:40:46 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1472804584592_0002/
    16/09/02 16:40:46 INFO mapreduce.Job: Running job: job_1472804584592_0002
    16/09/02 16:40:55 INFO mapreduce.Job: Job job_1472804584592_0002 running in uber mode : false
    16/09/02 16:40:55 INFO mapreduce.Job:  map 0% reduce 0%
    16/09/02 16:41:16 INFO mapreduce.Job:  map 2% reduce 0%
    16/09/02 16:41:31 INFO mapreduce.Job:  map 3% reduce 0%
    16/09/02 16:41:32 INFO mapreduce.Job:  map 4% reduce 0%
    16/09/02 16:41:44 INFO mapreduce.Job:  map 5% reduce 0%
    16/09/02 16:41:46 INFO mapreduce.Job:  map 6% reduce 0%
    16/09/02 16:41:59 INFO mapreduce.Job:  map 7% reduce 0%
    16/09/02 16:42:00 INFO mapreduce.Job:  map 8% reduce 0%
    16/09/02 16:42:12 INFO mapreduce.Job:  map 9% reduce 0%
    16/09/02 16:42:13 INFO mapreduce.Job:  map 10% reduce 0%
    16/09/02 16:42:26 INFO mapreduce.Job:  map 11% reduce 0%
    16/09/02 16:42:27 INFO mapreduce.Job:  map 12% reduce 0%
    16/09/02 16:42:40 INFO mapreduce.Job:  map 13% reduce 0%
    16/09/02 16:42:41 INFO mapreduce.Job:  map 14% reduce 0%
    16/09/02 16:42:55 INFO mapreduce.Job:  map 15% reduce 0%
    16/09/02 16:42:56 INFO mapreduce.Job:  map 16% reduce 0%
    16/09/02 16:43:10 INFO mapreduce.Job:  map 17% reduce 0%
    16/09/02 16:43:11 INFO mapreduce.Job:  map 18% reduce 0%
    16/09/02 16:43:25 INFO mapreduce.Job:  map 19% reduce 0%
    16/09/02 16:43:26 INFO mapreduce.Job:  map 20% reduce 0%
    16/09/02 16:43:39 INFO mapreduce.Job:  map 21% reduce 0%
    16/09/02 16:43:40 INFO mapreduce.Job:  map 22% reduce 0%
    16/09/02 16:43:52 INFO mapreduce.Job:  map 23% reduce 0%
    16/09/02 16:43:53 INFO mapreduce.Job:  map 24% reduce 0%
    16/09/02 16:44:06 INFO mapreduce.Job:  map 25% reduce 0%
    16/09/02 16:44:07 INFO mapreduce.Job:  map 26% reduce 0%
    16/09/02 16:44:21 INFO mapreduce.Job:  map 27% reduce 0%
    16/09/02 16:44:23 INFO mapreduce.Job:  map 28% reduce 0%
    16/09/02 16:44:35 INFO mapreduce.Job:  map 29% reduce 0%
    16/09/02 16:44:36 INFO mapreduce.Job:  map 30% reduce 0%
    16/09/02 16:44:48 INFO mapreduce.Job:  map 31% reduce 0%
    16/09/02 16:44:49 INFO mapreduce.Job:  map 32% reduce 0%
    16/09/02 16:44:59 INFO mapreduce.Job:  map 33% reduce 0%
    16/09/02 16:45:00 INFO mapreduce.Job:  map 34% reduce 0%
    16/09/02 16:45:11 INFO mapreduce.Job:  map 35% reduce 0%
    16/09/02 16:45:12 INFO mapreduce.Job:  map 36% reduce 0%
    16/09/02 16:45:22 INFO mapreduce.Job:  map 37% reduce 0%
    16/09/02 16:45:24 INFO mapreduce.Job:  map 38% reduce 0%
    16/09/02 16:45:35 INFO mapreduce.Job:  map 39% reduce 0%
    16/09/02 16:45:36 INFO mapreduce.Job:  map 40% reduce 0%
    16/09/02 16:45:46 INFO mapreduce.Job:  map 41% reduce 0%
    16/09/02 16:45:48 INFO mapreduce.Job:  map 42% reduce 0%
    16/09/02 16:45:58 INFO mapreduce.Job:  map 43% reduce 0%
    16/09/02 16:46:00 INFO mapreduce.Job:  map 44% reduce 0%
    16/09/02 16:46:12 INFO mapreduce.Job:  map 45% reduce 0%
    16/09/02 16:46:13 INFO mapreduce.Job:  map 46% reduce 0%
    16/09/02 16:46:23 INFO mapreduce.Job:  map 47% reduce 0%
    16/09/02 16:46:24 INFO mapreduce.Job:  map 48% reduce 0%
    16/09/02 16:46:34 INFO mapreduce.Job:  map 49% reduce 0%
    16/09/02 16:46:35 INFO mapreduce.Job:  map 50% reduce 0%
    16/09/02 16:46:45 INFO mapreduce.Job:  map 51% reduce 0%
    16/09/02 16:46:46 INFO mapreduce.Job:  map 52% reduce 0%
    16/09/02 16:46:59 INFO mapreduce.Job:  map 53% reduce 0%
    16/09/02 16:47:03 INFO mapreduce.Job:  map 53% reduce 18%
    16/09/02 16:47:07 INFO mapreduce.Job:  map 54% reduce 18%
    16/09/02 16:47:14 INFO mapreduce.Job:  map 55% reduce 18%
    16/09/02 16:47:21 INFO mapreduce.Job:  map 56% reduce 18%
    16/09/02 16:47:25 INFO mapreduce.Job:  map 56% reduce 19%
    16/09/02 16:47:28 INFO mapreduce.Job:  map 57% reduce 19%
    16/09/02 16:47:35 INFO mapreduce.Job:  map 58% reduce 19%
    16/09/02 16:47:42 INFO mapreduce.Job:  map 59% reduce 19%
    16/09/02 16:47:43 INFO mapreduce.Job:  map 59% reduce 20%
    16/09/02 16:47:49 INFO mapreduce.Job:  map 60% reduce 20%
    16/09/02 16:47:57 INFO mapreduce.Job:  map 61% reduce 20%
    16/09/02 16:48:05 INFO mapreduce.Job:  map 62% reduce 20%
    16/09/02 16:48:08 INFO mapreduce.Job:  map 62% reduce 21%
    16/09/02 16:48:14 INFO mapreduce.Job:  map 63% reduce 21%
    16/09/02 16:48:22 INFO mapreduce.Job:  map 64% reduce 21%
    16/09/02 16:48:31 INFO mapreduce.Job:  map 65% reduce 21%
    16/09/02 16:48:32 INFO mapreduce.Job:  map 65% reduce 22%
    16/09/02 16:48:41 INFO mapreduce.Job:  map 66% reduce 22%
    16/09/02 16:48:49 INFO mapreduce.Job:  map 67% reduce 22%
    16/09/02 16:48:57 INFO mapreduce.Job:  map 68% reduce 22%
    16/09/02 16:49:00 INFO mapreduce.Job:  map 68% reduce 23%
    16/09/02 16:49:05 INFO mapreduce.Job:  map 69% reduce 23%
    16/09/02 16:49:12 INFO mapreduce.Job:  map 70% reduce 23%
    16/09/02 16:49:20 INFO mapreduce.Job:  map 71% reduce 23%
    16/09/02 16:49:22 INFO mapreduce.Job:  map 71% reduce 24%
    16/09/02 16:49:28 INFO mapreduce.Job:  map 72% reduce 24%
    16/09/02 16:49:36 INFO mapreduce.Job:  map 73% reduce 24%
    16/09/02 16:49:43 INFO mapreduce.Job:  map 74% reduce 24%
    16/09/02 16:49:46 INFO mapreduce.Job:  map 74% reduce 25%
    16/09/02 16:49:50 INFO mapreduce.Job:  map 75% reduce 25%
    16/09/02 16:49:58 INFO mapreduce.Job:  map 76% reduce 25%
    16/09/02 16:50:09 INFO mapreduce.Job:  map 77% reduce 25%
    16/09/02 16:50:11 INFO mapreduce.Job:  map 77% reduce 26%
    16/09/02 16:50:17 INFO mapreduce.Job:  map 78% reduce 26%
    16/09/02 16:50:25 INFO mapreduce.Job:  map 79% reduce 26%
    16/09/02 16:50:32 INFO mapreduce.Job:  map 80% reduce 26%
    16/09/02 16:50:35 INFO mapreduce.Job:  map 80% reduce 27%
    16/09/02 16:50:39 INFO mapreduce.Job:  map 81% reduce 27%
    16/09/02 16:50:47 INFO mapreduce.Job:  map 82% reduce 27%
    16/09/02 16:50:55 INFO mapreduce.Job:  map 83% reduce 27%
    16/09/02 16:50:56 INFO mapreduce.Job:  map 83% reduce 28%
    16/09/02 16:51:03 INFO mapreduce.Job:  map 84% reduce 28%
    16/09/02 16:51:10 INFO mapreduce.Job:  map 85% reduce 28%
    16/09/02 16:51:17 INFO mapreduce.Job:  map 86% reduce 28%
    16/09/02 16:51:20 INFO mapreduce.Job:  map 86% reduce 29%
    16/09/02 16:51:25 INFO mapreduce.Job:  map 87% reduce 29%
    16/09/02 16:51:34 INFO mapreduce.Job:  map 88% reduce 29%
    16/09/02 16:51:41 INFO mapreduce.Job:  map 89% reduce 29%
    16/09/02 16:51:44 INFO mapreduce.Job:  map 89% reduce 30%
    16/09/02 16:51:49 INFO mapreduce.Job:  map 90% reduce 30%
    16/09/02 16:51:56 INFO mapreduce.Job:  map 91% reduce 30%
    16/09/02 16:52:03 INFO mapreduce.Job:  map 92% reduce 30%
    16/09/02 16:52:06 INFO mapreduce.Job:  map 92% reduce 31%
    16/09/02 16:52:11 INFO mapreduce.Job:  map 93% reduce 31%
    16/09/02 16:52:18 INFO mapreduce.Job:  map 94% reduce 31%
    16/09/02 16:52:26 INFO mapreduce.Job:  map 95% reduce 31%
    16/09/02 16:52:27 INFO mapreduce.Job:  map 95% reduce 32%
    16/09/02 16:52:34 INFO mapreduce.Job:  map 96% reduce 32%
    16/09/02 16:52:41 INFO mapreduce.Job:  map 97% reduce 32%
    16/09/02 16:52:48 INFO mapreduce.Job:  map 98% reduce 32%
    16/09/02 16:52:52 INFO mapreduce.Job:  map 98% reduce 33%
    16/09/02 16:52:55 INFO mapreduce.Job:  map 99% reduce 33%
    16/09/02 16:53:02 INFO mapreduce.Job:  map 100% reduce 33%
    16/09/02 16:53:03 INFO mapreduce.Job:  map 100% reduce 100%
    16/09/02 16:53:04 INFO mapreduce.Job: Job job_1472804584592_0002 completed successfully
    16/09/02 16:53:04 INFO mapreduce.Job: Counters: 49
            File System Counters
                    FILE: Number of bytes read=2206
                    FILE: Number of bytes written=11703871
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=26890
                    HDFS: Number of bytes written=215
                    HDFS: Number of read operations=403
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=3
            Job Counters
                    Launched map tasks=100
                    Launched reduce tasks=1
                    Data-local map tasks=100
                    Total time spent by all maps in occupied slots (ms)=921440
                    Total time spent by all reduces in occupied slots (ms)=376555
                    Total time spent by all map tasks (ms)=921440
                    Total time spent by all reduce tasks (ms)=376555
                    Total vcore-seconds taken by all map tasks=921440
                    Total vcore-seconds taken by all reduce tasks=376555
                    Total megabyte-seconds taken by all map tasks=943554560
                    Total megabyte-seconds taken by all reduce tasks=385592320
            Map-Reduce Framework
                    Map input records=100
                    Map output records=200
                    Map output bytes=1800
                    Map output materialized bytes=2800
                    Input split bytes=15090
                    Combine input records=0
                    Combine output records=0
                    Reduce input groups=2
                    Reduce shuffle bytes=2800
                    Reduce input records=200
                    Reduce output records=0
                    Spilled Records=400
                    Shuffled Maps =100
                    Failed Shuffles=0
                    Merged Map outputs=100
                    GC time elapsed (ms)=12309
                    CPU time spent (ms)=75150
                    Physical memory (bytes) snapshot=20894449664
                    Virtual memory (bytes) snapshot=86424981504
                    Total committed heap usage (bytes)=13431619584
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters
                    Bytes Read=11800
            File Output Format Counters
                    Bytes Written=97
    Job Finished in 741.887 seconds
    Estimated value of Pi is 3.14080000000000000000
    

    • 如果提示 copyFromLocal: Cannot create directory /123/. Name node is in safe mode.
      这是因为开启了安全模式(先关闭再启动Hadoop集群时,也会导致安全模式,此时不妨试试关闭安全模式)

      • 解决方法:
        cd /usr/local/hadoop
        bin/hdfs dfsadmin -safemode leave
  • 相关阅读:
    Pytest --快速入门和基础讲解
    Python并发编程(一):多进程(理论篇)
    【面试出题】三数之和
    python 基础:迭代器、生成器
    python垃圾回收机制
    50道Python面试题集锦
    Web自动化测试框架改进
    linux 三剑客
    函数的参数
    生成器
  • 原文地址:https://www.cnblogs.com/Genesis2018/p/8304728.html
Copyright © 2020-2023  润新知