• Spark运行模式:cluster与client


     

    When run SparkSubmit --class [mainClass], SparkSubmit will call a childMainClass which is

    1. client mode, childMainClass = mainClass

    2. standalone cluster mde, childMainClass = org.apache.spark.deploy.Client

    3. yarn cluster mode, childMainClass = org.apache.spark.deploy.yarn.Client

    The childMainClass is a wrapper of mainClass. The childMainClass will be called in SparkSubmit, and if cluster mode, the childMainClass will talk to the the cluster and launch a process on one woker to run the mainClass.
     
    ps. use "spark-submit -v" to print debug infos.
     
    Yarn client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn JavaWordCount.jar
    childMainclass: org.apache.spark.examples.JavaWordCount
    Yarn cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn-cluster JavaWordCount.jar
    childMainclass: org.apache.spark.deploy.yarn.Client
     
    Standalone client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 JavaWordCount.jar
    childMainclass: org.apache.spark.examples.JavaWordCount
    Stanalone cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 --deploy-mode cluster JavaWordCount.jar
    childMainclass: org.apache.spark.deploy.rest.RestSubmissionClient (if rest, else org.apache.spark.deploy.Client)
     
    Taking standalone spark as example, here is the client mode workflow. The mainclass run in the driver application which could be reside out of the cluster.
  • 相关阅读:
    给右键 添加dos命令
    js模拟系统无刷新跳回登录页1
    MBProgressHUD.h file not found
    建立个人博客网站
    <转>提高iOS开发效率的方法和工具
    设置模块功能设计思路及代码实现
    经验之谈
    'NSInteger' (aka 'long') to 'int32
    OCR技术
    升级Xcode6.4插件失效解决办法
  • 原文地址:https://www.cnblogs.com/c-abc/p/7457430.html
Copyright © 2020-2023  润新知