• hadoop 2012.07.24 读写测试


    namenode 独立虚拟机:1G内存

    datanode1、datanode2 虚拟于同一塔式服务器,内存各1G

     

    ----- TestDFSIO ----- : write
    Date & time: Tue Jul 24 17:15:06 CST 2012
    Number of files: 10
    Total MBytes processed: 10000
    Throughput mb/sec: 16.458439970308973
    Average IO rate mb/sec: 17.915302276611328
    IO rate std deviation: 5.502547439324805
    Test exec time sec: 702.77

    关闭数据库后测试

    Date & time: Tue Jul 24 18:16:42 CST 2012
    Number of files: 10
    Total MBytes processed: 10000
    Throughput mb/sec: 22.132315836999922
    Average IO rate mb/sec: 24.76445960998535
    IO rate std deviation: 8.614986952186385
    Test exec time sec: 543.453


    两台虚拟机作为datanode部署在一个节点上,在做写入测试时报Too many fetch-failures错误,这是因为读写过于频繁导致datanode与namenode通信异常导致的,移除其中一个datanode后再进行读写测试,就不再报联系不上的错误了。

    移除一台datanode后测试结果:

    ----- TestDFSIO ----- : write
    Date & time: Tue Jul 24 18:31:06 CST 2012
    Number of files: 10
    Total MBytes processed: 10000
    Throughput mb/sec: 14.386874566595404
    Average IO rate mb/sec: 14.926310539245605
    IO rate std deviation: 2.5935396939344066
    Test exec time sec: 456.993

    两台建立与独立机器上的虚拟机,其中有一个datanode没有配置hosts文件,导致DiskErrorException: Could not find taskTracker

    ----- TestDFSIO ----- : write
    Date & time: Thu Jul 26 01:00:34 PDT 2012
    Number of files: 10
    Total MBytes processed: 10000
    Throughput mb/sec: 54.977376809442916
    Average IO rate mb/sec: 64.9064712524414

    IO rate std deviation: 27.569772802264755
    Test exec time sec: 346.016

    两台建立与独立机器上的虚拟机,正常配置datanode的hosts文件后

    ----- TestDFSIO ----- : write
    Date & time: Thu Jul 26 16:28:38 CST 2012
    Number of files: 10
    Total MBytes processed: 10000
    Throughput mb/sec: 69.33322240017749
    Average IO rate mb/sec: 80.34981536865234
    IO rate std deviation: 31.796870275012598
    Test exec time sec: 333.343

  • 相关阅读:
    深入理解委托、匿名方法和 Lambda 表达式
    常见SQL问题
    LeetCode题解——四数之和
    把中台说清楚
    程序员们的三高:高并发、高性能、高可用
    论文查重是怎么查的
    LeetCode题解——最长回文子串
    六百字读懂 Git(转)
    SQL中ON和WHERE的区别
    链表排序之堆排序
  • 原文地址:https://www.cnblogs.com/MorZe/p/2606889.html
Copyright © 2020-2023  润新知