• Spark-shell 无法启动之网络问题


    由于需要首次手动安装sbt,需要联网,故将虚拟机的网络适配器模式设置为"桥接模式",这样就可以和互联网相连接。

    但是后面执行"spark-shell  --master yarn --deploy-mode client" 命令时,无法启动,一直停留在中间状态不动,

    如下:

    [root@master test_code]# spark-shell --master yarn --deploy-mode client
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    17/05/07 18:07:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platfor

    m... using builtin-java classes where applicable

    后来突然想起来,虚拟机的网络适配器模式没有更改过来,重新设置为"仅主机模式" 后(这是由于,安装Hadoop集群

    时,几个VMware虚拟机都是使用的"仅主机模式"),spark-shell 正常启动,如下:

    [root@master master]# spark-shell --master yarn --deploy-mode client
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    17/05/07 18:30:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/05/07 18:30:28 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
    17/05/07 18:31:15 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
    Spark context Web UI available at http://192.168.1.200:4040
    Spark context available as 'sc' (master = yarn, app id = application_1494142860645_0001).
    Spark session available as 'spark'.
    Welcome to
    ____ __
    / __/__ ___ _____/ /__
    _ / _ / _ `/ __/ '_/
    /___/ .__/\_,_/_/ /_/\_ version 2.1.0
    /_/

    Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
    Type in expressions to have them evaluated.
    Type :help for more information.

    scala>

    BTW:

    通过查看active NameNode 状态,可以看到,集群状态不正常,无法与其他节点通信,造成block丢失。如下WebUI 上所示:

    There are 35 missing blocks. The following files may be corrupted:

    blk_1073741947 /opt/hadoop/out_wordcount4/part-r-00000
    blk_1073741946 /opt/hadoop/out_wordcount2/part-r-00000
    blk_1073741945 /opt/hadoop/out_wordcount/part-r-00000
    blk_1073741933 /opt/hadoop/input/README.txt
    blk_1073741931 /hbase/MasterProcWALs/state-00000000000000000018.log
    blk_1073741930 /hbase/oldWALs/slave4%2C16020%2C1489814171196.meta.1489821379742.meta
    blk_1073741929 /hbase/oldWALs/slave3%2C16020%2C1489814171011.1489821375778
    blk_1073741928 /hbase/oldWALs/slave4%2C16020%2C1489814171196.1489821375970
    blk_1073741927 /hbase/oldWALs/slave5%2C16020%2C1489814170009.1489821374298
    blk_1073741920 /hbase/data/hbase/meta/1588230740/info/8e011b40156f4eeab4e83caf63ee1d23
    blk_1073741847 /hbase/data/hbase/namespace/3792ee8c4881d96201d73a19d76aa598/info/cc6c271a546248419df4c0988d191b4d
    blk_1073741846 /hbase/data/hbase/namespace/3792ee8c4881d96201d73a19d76aa598/.regioninfo
    blk_1073741845 /hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001
    blk_1073741968 /linkage/block_9.csv
    blk_1073741967 /linkage/block_8.csv
    blk_1073741839 /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
    blk_1073741966 /linkage/block_7.csv
    blk_1073741838 /hbase/data/hbase/meta/1588230740/.regioninfo
    blk_1073741837 /hbase/hbase.id
    blk_1073741965 /linkage/block_6.csv
    blk_1073741836 /hbase/hbase.version
    blk_1073741964 /linkage/block_5.csv
    blk_1073741963 /linkage/block_4.csv
    blk_1073741962 /linkage/block_3.csv
    blk_1073741961 /linkage/block_2.csv
    blk_1073741833 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1489568197327_0001.summary
    blk_1073741960 /linkage/block_1.csv
    blk_1073741832 /out/part-r-00000
    blk_1073741959 /linkage/block_10.csv
    blk_1073741958 /sogou/SogouQ1.txt
    blk_1073741957 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001_conf.xml
    blk_1073741956 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001-1492443717631-root-word+count-1492443749993-1-1-SUCCEEDED-default-1492443728809.jhist
    blk_1073741955 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1492443686126_0001.summary
    blk_1073741954 /opt/hadoop/out_1/part-r-00000
    blk_1073741825 /word

    Please check the logs or run fsck in order to identify the missing blocks. See the Hadoop FAQ for common causes and potential solutions.

  • 相关阅读:
    八、比卦
    七、师卦
    六、讼卦
    五、需卦
    力扣-两数之和
    什么是3NF (范式) ?
    SQL事务4个特性
    什么是索引?
    假设把只包含01的数组(如{0,0,1,1,1,0,1,0,0,1})按照升序排序,可以任意交换两个数的位置,请输出最少需要交换的次数。
    找规律并用编程实现如下数列(数值超过10000停止打印) 1,1,2,2,3,2,5,4,8,8
  • 原文地址:https://www.cnblogs.com/followyourdream/p/6821751.html
Copyright © 2020-2023  润新知