Storm 配置图文解析
參考阅读:http://www.xiaofateng.com/?
p=959
============================== | sample-topology | | ------------------------ | Task 1 Task 2 Task 3 | | Worker Process 1 | | T1 T2 T3 | | +--------+ | | Spout => Bolt => Bolt | | +------+ | +----+ | | | parallelism parallelism parallelism | | | T3 | | | T2 | | | | hint=2 hint=2 hint=6 | | +------+ | +----+ | | | | | | +----+ | | | combined parallelism = 2 + 2 + 6 = 10 | | +------+ | | T2 | | | | | | | T3 | | +----+ | | | Each of the 2 worker processes will spawn 10/2=5 threads | | +------+ +--------+ | | | | | | T1: parallelism hint = initial executors | | +------+ +--------+ | | | | | T3 | | T1 | | | T2: the T2 bolt was configured to use 2 executors and four tasks. | | +------+ +--------+ | | For this reason each executor runs two tasks for this bolt. | ------------------------ | | | | ------------------------ | Config conf = new Config(); | | Worker Process 2 | | | | +--------+ | | // run as 2 workers on 2 supervisors | | +------+ | +----+ | | | conf.setNumWorkers(2); | | | T3 | | | T2 | | | | | | +------+ | +----+ | | | // T1: 2 executors for spout | | | +----+ | | | topologyBuilder.setSpout("T1-spout", new T1Spout(), 2); | | +------+ | | T2 | | | | | | | T3 | | +----+ | | | // T2: 2 executors for bolt with total 4 tasks | | +------+ +--------+ | | topologyBuilder.setBolt("T2-bolt", new T2Bolt(), 2) | | | | .setNumTasks(4).shuffleGrouping("T1-spout"); | | +------+ +--------+ | | // T3: 6 executors for bolt (default 1 task for 1 executor) | | | T3 | | T1 | | | topologyBuilder.setBolt("T3-bolt", new T3Bolt(), 6).shuffleGrouping("T2-bolt"); | | +------+ +--------+ | | | ------------------------ | StormSubmitter.submitTopology("sample-topology", conf, topologyBuilder.createTopology()); ==============================
说明:
一个worker进程(process)会产生N个线程(executor),那么并行度(parallelism)就是全部线程的数目。setNumWorkers
任务(task)是线程运行的工作队列。线程的任务数说明线程的吞吐能力。一个线程的各个任务之间并非并发的。
setNumTasks
线程(executor)是运行任务的上下文环境。
參照上图理解各个配置的含义。