• Spark-shell 启动WARN---java.net.BindException: Address already in use


    同时打开了两个SecureCRT的终端界面,其中一个已经进入了Spark-shell,在另一个SecureCRT界面中执行

    "spark-shell --master yarn --deploy-mode client < cha02.scala" 时报错,如下:

    [root@master test_code]# spark-shell --master yarn --deploy-mode client <cha02.scala
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    17/05/07 20:41:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/05/07 20:41:37 WARN component.AbstractLifeCycle: FAILED ServerConnector@1953bc95{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)

    17/05/07 21:00:46 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
    17/05/07 21:00:59 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
    17/05/07 21:01:37 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
    Spark context Web UI available at http://192.168.1.200:4041
    Spark context available as 'sc' (master = yarn, app id = application_1494142860645_0005).
    Spark session available as 'spark'.
    Welcome to
    ____ __
    / __/__ ___ _____/ /__
    _ / _ / _ `/ __/ '_/
    /___/ .__/\_,_/_/ /_/\_ version 2.1.0
    /_/

    Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
    Type in expressions to have them evaluated.
    Type :help for more information.

    scala>

    这是由于启动一个Spark context 时,SparkUI 默认会使用 4040 端口,当 4040 端口被占用时,则尝试使用另外一个端口,即日志中的 

    Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

  • 相关阅读:
    Redis 安全
    Redis 数据备份与恢复
    Redis 服务器
    Redis 连接
    Redis 脚本
    Linux中使用netstat命令的基本操作,排查端口号的占用情况
    ElasticSearch 常用查询语句
    GO代码风格指南 Uber Go (转载)
    coding 注意事项(总结中)
    Byte字节
  • 原文地址:https://www.cnblogs.com/followyourdream/p/6822308.html
Copyright © 2020-2023  润新知