• Storm 配置图文解析


    Storm 配置图文解析

    參考阅读:http://www.xiaofateng.com/?

    p=959

         ==============================
         |     sample-topology        |
         |  ------------------------  |   Task 1           Task 2          Task 3
         |  |   Worker Process 1   |  |     T1               T2              T3
         |  |           +--------+ |  |    Spout     =>     Bolt      =>    Bolt
         |  | +------+  | +----+ | |  | parallelism      parallelism     parallelism
         |  | |  T3  |  | | T2 | | |  |    hint=2          hint=2          hint=6
         |  | +------+  | +----+ | |  |
         |  |           | +----+ | |  |    combined parallelism = 2 + 2 + 6 = 10
         |  | +------+  | | T2 | | |  |
         |  | |  T3  |  | +----+ | |  |  Each of the 2 worker processes will spawn 10/2=5 threads
         |  | +------+  +--------+ |  |
         |  |                      |  |    T1: parallelism hint = initial executors
         |  | +------+  +--------+ |  |
         |  | |  T3  |  |   T1   | |  |    T2: the T2 bolt was configured to use 2 executors and four tasks.
         |  | +------+  +--------+ |  |        For this reason each executor runs two tasks for this bolt.
         |  ------------------------  |
         |                            |
         |  ------------------------  |    Config conf = new Config();
         |  |   Worker Process 2   |  |
         |  |           +--------+ |  |    // run as 2 workers on 2 supervisors
         |  | +------+  | +----+ | |  |    conf.setNumWorkers(2);
         |  | |  T3  |  | | T2 | | |  |
         |  | +------+  | +----+ | |  |    // T1: 2 executors for spout
         |  |           | +----+ | |  |    topologyBuilder.setSpout("T1-spout", new T1Spout(), 2);
         |  | +------+  | | T2 | | |  |
         |  | |  T3  |  | +----+ | |  |    // T2: 2 executors for bolt with total 4 tasks
         |  | +------+  +--------+ |  |    topologyBuilder.setBolt("T2-bolt", new T2Bolt(), 2)
         |  |                      |  |        .setNumTasks(4).shuffleGrouping("T1-spout");
         |  | +------+  +--------+ |  |    // T3: 6 executors for bolt (default 1 task for 1 executor)
         |  | |  T3  |  |   T1   | |  |    topologyBuilder.setBolt("T3-bolt", new T3Bolt(), 6).shuffleGrouping("T2-bolt");
         |  | +------+  +--------+ |  |
         |  ------------------------  |    StormSubmitter.submitTopology("sample-topology", conf, topologyBuilder.createTopology());
         ==============================
    

    说明:

    一个worker进程(process)会产生N个线程(executor),那么并行度(parallelism)就是全部线程的数目。setNumWorkers

    任务(task)是线程运行的工作队列。线程的任务数说明线程的吞吐能力。一个线程的各个任务之间并非并发的。

    setNumTasks

    线程(executor)是运行任务的上下文环境。

    參照上图理解各个配置的含义。




  • 相关阅读:
    DLL导出类避免地狱问题的完美解决方案
    WorldWind源码剖析系列:影像图层类ImageLayer
    WorldWind源码剖析系列:插件类Plugin、插件信息类PluginInfo和插件编译器类PluginCompiler
    多线程中的锁系统(一)-基础用法
    [转]分布式计算框架综述
    C#自定义控件开发
    GDI+编程小结
    C#自定义控件
    WorldWind源码剖析系列:窗口定制控件类WorldWindow
    WorldWind源码剖析系列:四叉树瓦片集合类QuadTileSet
  • 原文地址:https://www.cnblogs.com/bhlsheji/p/5215876.html
Copyright © 2020-2023  润新知