《Apache kafka实战》读书笔记-kafka集群监控工具
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
如官网所述，Kafka使用基于yammer metrics的监控指标体系来统计broker端和clinets端的各种监控指标（metric）。说到yammer metrics，其官网过出了这样的一句话：
yammer metrics是一个java库，它使得你能够对生产环境代码所做之事具有无与伦比的洞察力。
当然，虽然该项目已经进化到3.x版本，但即使最新版本的kafka易燃在使用2.2版本的yammer metrics。目前，Kafka默认提供了超多的监控指标，无一例外，用户皆使用JMX接口访问这些指标。JMX，即Java Management Extension，Java管理扩展，是一套为各个应用程序，设备或系统等植入管理功能的框架。JMX本身是跨平台的，因此具有高度的灵活性，可以无缝集成进各种系统中。
虽然有一小部分功能是以Shell脚本直接允许的Java类来提供的，但是Kafka集群大部分的允许表现都可由JMX指标来表征，因此搭建一套能够访问JMX的监控框架或系统对于监控kafka集群而言至关重要。
Kafka的每个监控指标都是以JMX MBean的形式定义的。虽然JMX规范不再本书的讨论范围内，但这里依然简单介绍一些MBean的概念。MBean代表一个被管理的资源实例，他表示管理资源的Java对象。每个MBean的管理接口如下：
1>.属性值;
2>.能够执行的操作；
3>.能发出的通知事件；
4>.构建器；
对于kafka而言，我们只关系上述列表中的第一项，即MBean的属性值，它包含了Kafka各种监控指标的真实值。MBean需要注册之后才能使用，在注册时必须指定MBean的名称。在kafka中，所有MBean的命名规范有着统一的的格式：“xxx.type=xxx,{attr}=xxx”。第一个等式中的xxx通常表示所属的kafka组建，比如“kafka.server”,"kafka.producer","kafka.consumer"等；第二个等式的中的xxx和前面的attr通常表示该MBean的范围，比如topic=yinzhengjie-kafka表示该MBean的作用范围是名为“yinzhengjie-kafka”的topic，即它是一个topic级别的MBean，表示该特topic下的某个性能指标。
对于Java用户而言，访问JMX端口最简单直接方法就是使用JDK自带的JConsole工具（可以参考我之前的笔记：https://www.cnblogs.com/yinzhengjie/p/9279283.html）。当然，你也可以直接查看本篇博客的末尾，我使用了jmxTool工具的一个案例。
一.安装Kafka Monitor
kafka Monitor 是LinkedIn公司开源的一个免费框架，用于在kafka集群上执行真实的kafka系统测试并实时监控测试度量结果。该框架由LinkedIn的kafka团队成员维护，维护的速度要比kafka-manager快一些。
如上所述，kafka monitor是在集群上执行系统测试的，以便让用户对Kafka集群的性能有大致的了解。他会自动创建一个监控topic，然后分配分区并触发preferred leader选举以确保每个broker都至少是一个分区的leader。这样做的好处在于可以全面地测试集群上所有broker的性能而无须用户手动干预。
1>.安装git工具包
[root@node101 ~]# yum -y install git Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile 10gen | 2.5 kB 00:00:00 base | 3.6 kB 00:00:00 centosplus | 3.4 kB 00:00:00 epel | 3.2 kB 00:00:00 extras | 3.4 kB 00:00:00 mysql-connectors-community | 2.5 kB 00:00:00 mysql-tools-community | 2.5 kB 00:00:00 mysql56-community | 2.5 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 934 kB 00:00:00 (2/2): epel/x86_64/primary | 3.6 MB 00:00:01 epel 12720/12720 Resolving Dependencies --> Running transaction check ---> Package git.x86_64 0:1.8.3.1-14.el7_5 will be installed --> Processing Dependency: perl-Git = 1.8.3.1-14.el7_5 for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Git) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Error) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: libgnome-keyring.so.0()(64bit) for package: git-1.8.3.1-14.el7_5.x86_64 --> Running transaction check ---> Package libgnome-keyring.x86_64 0:3.12.0-1.el7 will be installed ---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed ---> Package perl-Git.noarch 0:1.8.3.1-14.el7_5 will be installed ---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================== Installing: git x86_64 1.8.3.1-14.el7_5 updates 4.4 M Installing for dependencies: libgnome-keyring x86_64 3.12.0-1.el7 base 109 k perl-Error noarch 1:0.17020-2.el7 base 32 k perl-Git noarch 1.8.3.1-14.el7_5 updates 54 k perl-TermReadKey x86_64 2.30-20.el7 base 31 k Transaction Summary =================================================================================================================================================================================== Install 1 Package (+4 Dependent packages) Total download size: 4.6 M Installed size: 23 M Downloading packages: (1/5): perl-Error-0.17020-2.el7.noarch.rpm | 32 kB 00:00:00 (2/5): libgnome-keyring-3.12.0-1.el7.x86_64.rpm | 109 kB 00:00:00 (3/5): perl-TermReadKey-2.30-20.el7.x86_64.rpm | 31 kB 00:00:00 (4/5): perl-Git-1.8.3.1-14.el7_5.noarch.rpm | 54 kB 00:00:00 (5/5): git-1.8.3.1-14.el7_5.x86_64.rpm | 4.4 MB 00:00:00 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 19 MB/s | 4.6 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:perl-Error-0.17020-2.el7.noarch 1/5 Installing : perl-TermReadKey-2.30-20.el7.x86_64 2/5 Installing : libgnome-keyring-3.12.0-1.el7.x86_64 3/5 Installing : git-1.8.3.1-14.el7_5.x86_64 4/5 Installing : perl-Git-1.8.3.1-14.el7_5.noarch 5/5 Verifying : 1:perl-Error-0.17020-2.el7.noarch 1/5 Verifying : git-1.8.3.1-14.el7_5.x86_64 2/5 Verifying : libgnome-keyring-3.12.0-1.el7.x86_64 3/5 Verifying : perl-Git-1.8.3.1-14.el7_5.noarch 4/5 Verifying : perl-TermReadKey-2.30-20.el7.x86_64 5/5 Installed: git.x86_64 0:1.8.3.1-14.el7_5 Dependency Installed: libgnome-keyring.x86_64 0:3.12.0-1.el7 perl-Error.noarch 1:0.17020-2.el7 perl-Git.noarch 0:1.8.3.1-14.el7_5 perl-TermReadKey.x86_64 0:2.30-20.el7 Complete! [root@node101 ~]#
2>.克隆kafka-monitor
[root@node101 ~]# cd /yinzhengjie/kafka-monitor/ [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# ll total 0 [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# git clone https://github.com/linkedin/kafka-monitor.git Cloning into 'kafka-monitor'... remote: Enumerating objects: 16, done. remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 100% (12/12), done. remote: Total 1199 (delta 1), reused 7 (delta 0), pack-reused 1183 Receiving objects: 100% (1199/1199), 583.88 KiB | 248.00 KiB/s, done. Resolving deltas: 100% (479/479), done. [root@node101 kafka-monitor]# [root@node101 kafka-monitor]# ll total 0 drwxr-xr-x 11 root root 309 Dec 3 17:38 kafka-monitor [root@node101 kafka-monitor]#
3>.构建gradlew
[root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 17:35 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# ./gradlew jar
下面是构建时他会自动联网下载一些maven依赖关系：
如果你安装成功的话，说明问题不大， 但是你的Java环境如果没有部署好的话，可能会抛异常如下：
..................(一系列消息被我省略) 18:12:29.045 [INFO] [org.gradle.api.internal.tasks.compile.JdkJavaCompiler] Compiling with JDK Java compiler API. 18:12:29.045 [DEBUG] [org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter] Removed task artifact state for {} from context. 18:12:29.045 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':compileJava' 18:12:29.046 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] :compileJava FAILED 18:12:29.046 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :compileJava (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.06 secs. 18:12:29.046 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationWorkerRegistry] Worker root.1 completed (0 in use) 18:12:29.046 [DEBUG] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] Task worker [Thread[Daemon worker Thread 3,5,main]] finished, busy: 0.06 secs, idle: 0.0 secs 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 18:12:29.047 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':compileJava'. 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not find tools.jar. Please check that /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre contains a valid JDK installation. 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try: 18:12:29.048 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace. 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] BUILD FAILED 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] 18:12:29.048 [LIFECYCLE] [org.gradle.internal.buildevents.BuildResultLogger] Total time: 1.174 secs 18:12:29.060 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for task history cache (/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor/.gradle/3.2.1/taskArtifacts) 18:12:29.060 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on task history cache (/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor/.gradle/3.2.1/taskArtifacts). 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs) 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs) 18:12:29.064 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.001 secs 18:12:29.064 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Plugin Resolution Cache (/root/.gradle/caches/3.2.1/plugin-resolution) was closed 0 times. 18:12:29.064 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Generated Gradle JARs cache (/root/.gradle/caches/3.2.1/generated-gradle-jars) was closed 0 times. 18:12:29.064 [DEBUG] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopping 0 compiler daemon(s). 18:12:29.065 [INFO] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopped 0 compiler daemon(s). 18:12:29.065 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for artifact cache (/root/.gradle/caches/modules-2) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-metadata.bin (/root/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache artifact-at-repository.bin (/root/.gradle/caches/modules-2/metadata-2.23/artifact-at-repository.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-artifacts.bin (/root/.gradle/caches/modules-2/metadata-2.23/module-artifacts.bin) 18:12:29.065 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on artifact cache (/root/.gradle/caches/modules-2). 18:12:29.065 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build. 18:12:29.275 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientInputForwarder] Dispatching close input message: org.gradle.launcher.daemon.protocol.CloseInput@1f6ddd9f 18:12:29.275 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 14: dispatching class org.gradle.launcher.daemon.protocol.CloseInput 18:12:29.276 [INFO] [org.gradle.launcher.daemon.client.DaemonClient] Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: Execution failed for task ':compileJava'.] from daemon DaemonInfo{pid=18961, address=[d4fe0de0-9944-4ad9-a617-eae681b59f33 port:42738, addresses:[/127.0.0.1]], state=Idle, lastBusy=1543831940300, context=DefaultDaemonContext[uid=16910369-6f16-4b04-b799-3e7b93a3a0e6,javaHome=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre,daemonRegistryDir=/root/.gradle/daemon,pid=18961,idleTimeout=10800000,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done). 18:12:29.276 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished 18:12:29.277 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: connection stop [root@yinzhengjie kafka-monitor]#
报错分析：
根据上述报错信息，提示找不到tools.jar。可能是你的java是基于yum方式安装的，很多工具包可能不全。
解决方案：
将你下载好的jdk中存在的tools.jar包拷贝过去即可。
[root@yinzhengjie ~]# find / -name tools.jar /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar [root@yinzhengjie ~]# [root@yinzhengjie ~]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/lib/ [root@yinzhengjie ~]# [root@yinzhengjie ~]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/ [root@yinzhengjie ~]# [root@yinzhengjie ~]# find / -name tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/lib/tools.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/tools.jar /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/lib/tools.jar [root@yinzhengjie ~]#
将tools.jar包拷贝到指定环境变量后，我们开始执行以下操作：（你会发现变异成功啦！）
[root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 17:35 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# ./gradlew jar :compileJava UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :copyDependantLibs UP-TO-DATE :jar UP-TO-DATE BUILD SUCCESSFUL Total time: 1.073 secs [root@yinzhengjie kafka-monitor]#
4>.修改kafka-monitor.properties配置文件
[root@yinzhengjie kafka-monitor]# pwd /yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# cat config/kafka-monitor.properties | grep -v ^# | grep -v ^$ { "single-cluster-monitor": { "class.name": "com.linkedin.kmf.apps.SingleClusterMonitor", "topic": "kafka-monitor-topic", "zookeeper.connect": "10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181", #指定zookeeper地址 "bootstrap.servers": "10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092", #指定broker地址 "produce.record.delay.ms": 100, "topic-management.topicCreationEnabled": true, "topic-management.replicationFactor" : 1, "topic-management.partitionsToBrokersRatio" : 2.0, "topic-management.rebalance.interval.ms" : 600000, "topic-management.topicFactory.props": { }, "topic-management.topic.props": { "retention.ms": "3600000" }, "produce.producer.props": { "client.id": "kmf-client-id" }, "consume.latency.sla.ms": "20000", "consume.consumer.props": { } }, "jetty-service": { "class.name": "com.linkedin.kmf.services.JettyService", "jetty.port": 8000 }, "jolokia-service": { "class.name": "com.linkedin.kmf.services.JolokiaService" }, "reporter-service": { "class.name": "com.linkedin.kmf.services.DefaultMetricsReporterService", "report.interval.sec": 1, "report.metrics.list": [ "kmf:type=kafka-monitor:offline-runnable-count", "kmf.services:type=produce-service,name=*:produce-availability-avg", "kmf.services:type=consume-service,name=*:consume-availability-avg", "kmf.services:type=produce-service,name=*:records-produced-total", "kmf.services:type=consume-service,name=*:records-consumed-total", "kmf.services:type=consume-service,name=*:records-lost-total", "kmf.services:type=consume-service,name=*:records-duplicated-total", "kmf.services:type=consume-service,name=*:records-delay-ms-avg", "kmf.services:type=produce-service,name=*:records-produced-rate", "kmf.services:type=produce-service,name=*:produce-error-rate", "kmf.services:type=consume-service,name=*:consume-error-rate" ] } } [root@yinzhengjie kafka-monitor]#
5>.启动kafka monitor服务
[root@yinzhengjie kafka-monitor]# pwd /yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-monitor [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# ll total 64 drwxr-xr-x 3 root root 129 Dec 3 17:35 bin drwxr-xr-x 7 root root 85 Dec 3 18:14 build -rw-r--r-- 1 root root 1797 Dec 3 17:35 build.gradle drwxr-xr-x 2 root root 50 Dec 3 17:35 checkstyle drwxr-xr-x 2 root root 130 Dec 3 19:26 config -rw-r--r-- 1 root root 906 Dec 3 17:35 CONTRIBUTING.md drwxr-xr-x 2 root root 90 Dec 3 17:35 docker drwxr-xr-x 3 root root 19 Dec 3 17:35 docs drwxr-xr-x 3 root root 20 Dec 3 17:35 gradle -rwxr-xr-x 1 root root 4971 Dec 3 17:35 gradlew -rw-r--r-- 1 root root 2404 Dec 3 17:35 gradlew.bat -rw-r--r-- 1 root root 28832 Dec 3 17:35 LICENSE drwxr-xr-x 2 root root 29 Dec 3 19:24 logs -rw-r--r-- 1 root root 1371 Dec 3 17:35 NOTICE -rw-r--r-- 1 root root 6369 Dec 3 17:35 README.md -rw-r--r-- 1 root root 0 Dec 3 17:35 settings.gradle drwxr-xr-x 4 root root 28 Dec 3 17:35 src drwxr-xr-x 3 root root 32 Dec 3 17:35 webapp [root@yinzhengjie kafka-monitor]# yinzhengjie [root@yinzhengjie kafka-monitor]# mkdir -pv /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs mkdir: created directory ‘/yinzhengjie/softwares/kafkasoft/kafka-monitor/logs’ [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# bin/kafka-monitor-start.sh config/kafka-monitor.properties >> /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-`date +%F`.log & [1] 16888 [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# jobs [1]+ Running bin/kafka-monitor-start.sh config/kafka-monitor.properties >> /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-`date +%F`.log & [root@yinzhengjie kafka-monitor]# [root@yinzhengjie kafka-monitor]# tail -1000f /yinzhengjie/softwares/kafkasoft/kafka-monitor/logs/kafka-monitor-2018-12-03.log | more [2018-12-03 19:30:12,160] INFO MultiClusterTopicManagementServiceConfig values: topic = kafka-monitor-topic topic-management.rebalance.interval.ms = 600000 (com.linkedin.kmf.services.configs.MultiClusterTopicManagementServiceConfig) [2018-12-03 19:30:12,240] INFO TopicManagementServiceConfig values: topic = kafka-monitor-topic topic-management.minPartitionNum = 1 topic-management.partitionsToBrokersRatio = 2.0 topic-management.replicationFactor = 1 topic-management.topicCreationEnabled = true topic-management.topicFactory.class.name = com.linkedin.kmf.topicfactory.DefaultTopicFactory zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.TopicManagementServiceConfig) [2018-12-03 19:30:12,244] INFO ProduceServiceConfig values: bootstrap.servers = 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 produce.latency.percentile.granularity.ms = 1 produce.latency.percentile.max.ms = 5000 produce.partitioner.class = class com.linkedin.kmf.partitioner.NewKMPartitioner produce.producer.class = com.linkedin.kmf.producer.NewProducer produce.producer.id = kmf-producer produce.record.delay.ms = 100 produce.record.size.byte = 100 produce.sync = true produce.thread.num = 5 produce.treat.zero.throughput.as.unavailable = true topic = kafka-monitor-topic zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.ProduceServiceConfig) [2018-12-03 19:30:12,338] INFO single-cluster-monitor/ProduceService is initialized. (com.linkedin.kmf.services.ProduceService) [2018-12-03 19:30:12,347] INFO ConsumeServiceConfig values: bootstrap.servers = 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 consume.consumer.class = com.linkedin.kmf.consumer.NewConsumer consume.latency.percentile.granularity.ms = 1 consume.latency.percentile.max.ms = 5000 consume.latency.sla.ms = 20000 topic = kafka-monitor-topic zookeeper.connect = 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 (com.linkedin.kmf.services.configs.ConsumeServiceConfig) --More--
6>.检查服务是否启动成功
[root@yinzhengjie ~]# netstat -untalp | grep 8000 tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 16888/java [root@yinzhengjie ~]# [root@yinzhengjie ~]# hostname -i 10.19.104.189 [root@yinzhengjie ~]#
我们可以通过webUI的方式访问kafka-monitor服务：
Kafka Monitor作为LinkedIn Kafka 团队维护的开源框架，从系统测试的角度给出了kafka集群的性能评测结果。从这个意义上来说，它不同于其他的监控框架。且平台目前提供的Web界面功能十分简单，因此实际使用用户数并不是很多。不过鉴于其代码贡献者基本上都是Kafka社区团队成员，其框架的发展还是值得期待的。
二.安装Kafka Offset Monitor
这款监控工具不推荐大家使用，因为这个项目已经不维护了啦。尤其是kafka0.9.0.0以后的kafka版本，不建议使用，因为我用kafka0.10.2.1这个版本时，它是无法监控到broker集群的信息的！这一点很让人蛋疼！令人欣喜的是，该项目又一个同名的“姊妹”项目，地址是：“https://github.com/Morningstar/kafka-offset-monitor”，感兴趣的小伙伴可以捣鼓一下。
我已经下载好相应的安装包了，感兴趣的小伙伴可以直接拿去用，我给出百度云链接:https://pan.baidu.com/s/1Qbgtxtdd_1B2hlqatA0GRQ 密码:u991
[root@yinzhengjie kafka-offset-monitor]# ll total 54468 -rw-r--r-- 1 root root 52937155 Dec 3 20:00 KafkaOffsetMonitor-assembly-0.2.1.jar -rw-r--r-- 1 root root 2830336 Dec 3 20:38 offsetapp.db [root@yinzhengjie kafka-offset-monitor]# [root@yinzhengjie kafka-offset-monitor]# java -cp KafkaOffsetMonitor-assembly-0.2.1.jar:kafka-offset-monitor-another-db-reporter.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 10.1.3.117:2181,10.1.3.118:2181,10.1.3.119:2181 --port 1010 --refresh 10.seconds --retain 2.days serving resources from: jar:file:/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-offset-monitor/KafkaOffsetMonitor-assembly-0.2.1.jar!/offsetapp SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No appenders could be found for logger (org.I0Itec.zkclient.ZkConnection). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. log4j:WARN No appenders could be found for logger (org.I0Itec.zkclient.ZkEventThread). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 2018-12-03 20:08:56.101:INFO:oejs.Server:jetty-7.x.y-SNAPSHOT 2018-12-03 20:08:56.132:INFO:oejsh.ContextHandler:started o.e.j.s.ServletContextHandler{/,jar:file:/data/yinzhengjie/softwares/kafkasoft/kafka-monitor/kafka-offset-monitor/KafkaOffsetMonitor-assembly-0.2.1.jar!/offsetapp} 2018-12-03 20:08:56.139:INFO:oejs.AbstractConnector:Started SocketConnector@0.0.0.0:1010
然后你在webUI访问该主机的1010端口即可，如果你的kafka版本过高的话，估计够呛，你应该查不到相应的信息！
三.CruiseControl部署
我们聊聊LinkedIn公司于2017年8月底开源的Kafka监控框架，即CruiseControl。它主要用于管理查大规模的Kafka集群，提供的功能如下：
1>.实时监控资源使用率（broker，topic，分区）。
2>.提供多纬度的重平衡策略：
2.1>.机架感知（rack awareness）
2.2>.资源使用率再平衡（CPU，磁盘，网络I/O）
2.3>.leader副本流量分配
2.4>.副本均匀分布
3>.异常检测以及实时告警。
4>.开箱即用的运维操作：
4.1>.增加broker节点
4.2>.去除broker节点
4.3>.集群在平衡。
由于该框架刚刚开源，有很多功能尚在完善和演进中。打算使用CruiseControl的用户最好时刻关注该框架的发展，其官网地址是：https://github.com/linkedin/cruise-control。
接下来我们掩饰一些如何安装，配置和启动CruiseControl的详细步骤。
1>.下载源码
[root@yinzhengjie kafka-cruiseControl]# ll total 0 [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# git clone https://github.com/linkedin/cruise-control.git Cloning into 'cruise-control'... remote: Enumerating objects: 47, done. remote: Counting objects: 100% (47/47), done. remote: Compressing objects: 100% (32/32), done. remote: Total 7615 (delta 9), reused 16 (delta 0), pack-reused 7568 Receiving objects: 100% (7615/7615), 2.41 MiB | 735.00 KiB/s, done. Resolving deltas: 100% (3537/3537), done. [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# ll total 4 drwxr-xr-x 12 root root 4096 Dec 4 10:45 cruise-control [root@yinzhengjie kafka-cruiseControl]#
2>.进行编译
[root@yinzhengjie kafka-cruiseControl]# ll total 4 drwxr-xr-x 12 root root 4096 Dec 4 10:45 cruise-control [root@yinzhengjie kafka-cruiseControl]# [root@yinzhengjie kafka-cruiseControl]# cd cruise-control/ [root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 3 root root 35 Dec 4 10:45 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control-core drwxr-xr-x 3 root root 16 Dec 4 10:45 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew clean jar Downloading https://services.gradle.org/distributions/gradle-4.8-all.zip ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Unzipping /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl/gradle-4.8-all.zip to /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl Set executable permissions for: /root/.gradle/wrapper/dists/gradle-4.8-all/41c9mvq2w0645xx6nsenaerjl/gradle-4.8/bin/gradle Welcome to Gradle 4.8! Here are the highlights of this release: - Dependency locking - Maven Publish and Ivy Publish plugins improved and marked stable - Incremental annotation processing enhancements - APIs to configure tasks at creation time For more details see https://docs.gradle.org/4.8/release-notes.html To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing Download https://jcenter.bintray.com/org/apache/httpcomponents/fluent-hc/4.5.2/fluent-hc-4.5.2.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor-gradle/4.7.3/build-info-extractor-gradle-4.7.3.pom Download https://jcenter.bintray.com/org/ajoberstar/gradle-git/1.2.0/gradle-git-1.2.0.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-client/4.5.2/httpcomponents-client-4.5.2.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/project/7/project-7.pom Download https://jcenter.bintray.com/org/apache/apache/13/apache-13.pom Download https://jcenter.bintray.com/org/ajoberstar/grgit/1.3.0/grgit-1.3.0.pom Download https://jcenter.bintray.com/com/github/zafarkhaja/java-semver/0.9.0/java-semver-0.9.0.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/9/oss-parent-9.pom Download https://jcenter.bintray.com/commons-logging/commons-logging/1.2/commons-logging-1.2.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/34/commons-parent-34.pom Download https://jcenter.bintray.com/commons-lang/commons-lang/2.4/commons-lang-2.4.pom Download https://jcenter.bintray.com/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor/2.11.2/build-info-extractor-2.11.2.pom Download https://jcenter.bintray.com/org/apache/apache/7/apache-7.pom Download https://jcenter.bintray.com/commons-io/commons-io/2.2/commons-io-2.2.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/24/commons-parent-24.pom Download https://jcenter.bintray.com/org/apache/apache/9/apache-9.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit/3.7.0.201502260915-r/org.eclipse.jgit-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy/0.0.7/jsch.agentproxy-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit-parent/3.7.0.201502260915-r/org.eclipse.jgit-parent-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/6/oss-parent-6.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.pom Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit.ui/3.7.0.201502260915-r/org.eclipse.jgit.ui-3.7.0.201502260915-r.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.pom Download https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom Download https://jcenter.bintray.com/org/slf4j/slf4j-parent/1.7.7/slf4j-parent-1.7.7.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.pom Download https://jcenter.bintray.com/commons-codec/commons-codec/1.9/commons-codec-1.9.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-core/4.4.4/httpcomponents-core-4.4.4.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/32/commons-parent-32.pom Download https://jcenter.bintray.com/com/google/guava/guava/18.0/guava-18.0.pom Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream/1.4.10/xstream-1.4.10.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-client/2.11.2/build-info-client-2.11.2.pom Download https://jcenter.bintray.com/com/google/guava/guava-parent/18.0/guava-parent-18.0.pom Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream-parent/1.4.10/xstream-parent-1.4.10.pom Download https://jcenter.bintray.com/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.pom Download https://jcenter.bintray.com/com/jcraft/jsch/0.1.50/jsch-0.1.50.pom Download https://jcenter.bintray.com/org/sonatype/oss/oss-parent/5/oss-parent-5.pom Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.pom Download https://jcenter.bintray.com/net/java/dev/jna/platform/3.4.0/platform-3.4.0.pom Download https://jcenter.bintray.com/net/java/dev/jna/jna/3.4.0/jna-3.4.0.pom Download https://jcenter.bintray.com/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1.pom Download https://jcenter.bintray.com/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.5/httpcore-4.4.5.pom Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcomponents-core/4.4.5/httpcomponents-core-4.4.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.9.5/jackson-core-2.9.5.pom Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-api/2.11.2/build-info-api-2.11.2.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-base/2.9.5/jackson-base-2.9.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-bom/2.9.5/jackson-bom-2.9.5.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/jackson-parent/2.9.1/jackson-parent-2.9.1.pom Download https://jcenter.bintray.com/com/fasterxml/oss-parent/30/oss-parent-30.pom Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.9.5/jackson-databind-2.9.5.pom Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.16/commons-compress-1.16.pom Download https://jcenter.bintray.com/org/apache/commons/commons-parent/43/commons-parent-43.pom Download https://jcenter.bintray.com/org/apache/apache/18/apache-18.pom Download https://jcenter.bintray.com/org/objenesis/objenesis/2.6/objenesis-2.6.pom Download https://jcenter.bintray.com/org/objenesis/objenesis-parent/2.6/objenesis-parent-2.6.pom Generating JAR file 'gradle-api-4.8.jar' Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor-gradle/4.7.3/build-info-extractor-gradle-4.7.3.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/fluent-hc/4.5.2/fluent-hc-4.5.2.jar Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit.ui/3.7.0.201502260915-r/org.eclipse.jgit.ui-3.7.0.201502260915-r.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-client/2.11.2/build-info-client-2.11.2.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-extractor/2.11.2/build-info-extractor-2.11.2.jar Download https://jcenter.bintray.com/org/jfrog/buildinfo/build-info-api/2.11.2/build-info-api-2.11.2.jar Download https://jcenter.bintray.com/org/eclipse/jgit/org.eclipse.jgit/3.7.0.201502260915-r/org.eclipse.jgit-3.7.0.201502260915-r.jar Download https://jcenter.bintray.com/commons-logging/commons-logging/1.2/commons-logging-1.2.jar Download https://jcenter.bintray.com/org/ajoberstar/grgit/1.3.0/grgit-1.3.0.jar Download https://jcenter.bintray.com/commons-io/commons-io/2.2/commons-io-2.2.jar Download https://jcenter.bintray.com/org/ajoberstar/gradle-git/1.2.0/gradle-git-1.2.0.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.jsch/0.0.7/jsch.agentproxy.jsch-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.pageant/0.0.7/jsch.agentproxy.pageant-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.sshagent/0.0.7/jsch.agentproxy.sshagent-0.0.7.jar Download https://jcenter.bintray.com/commons-lang/commons-lang/2.4/commons-lang-2.4.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-jna/0.0.7/jsch.agentproxy.usocket-jna-0.0.7.jar Download https://jcenter.bintray.com/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.usocket-nc/0.0.7/jsch.agentproxy.usocket-nc-0.0.7.jar Download https://jcenter.bintray.com/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar Download https://jcenter.bintray.com/org/apache/httpcomponents/httpcore/4.4.5/httpcore-4.4.5.jar Download https://jcenter.bintray.com/commons-codec/commons-codec/1.9/commons-codec-1.9.jar Download https://jcenter.bintray.com/com/google/guava/guava/18.0/guava-18.0.jar Download https://jcenter.bintray.com/com/thoughtworks/xstream/xstream/1.4.10/xstream-1.4.10.jar Download https://jcenter.bintray.com/com/googlecode/javaewah/JavaEWAH/0.7.9/JavaEWAH-0.7.9.jar Download https://jcenter.bintray.com/com/jcraft/jsch.agentproxy.core/0.0.7/jsch.agentproxy.core-0.0.7.jar Download https://jcenter.bintray.com/com/jcraft/jsch/0.1.50/jsch-0.1.50.jar Download https://jcenter.bintray.com/net/java/dev/jna/platform/3.4.0/platform-3.4.0.jar Download https://jcenter.bintray.com/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1.jar Download https://jcenter.bintray.com/net/java/dev/jna/jna/3.4.0/jna-3.4.0.jar Download https://jcenter.bintray.com/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-databind/2.9.5/jackson-databind-2.9.5.jar Download https://jcenter.bintray.com/org/objenesis/objenesis/2.6/objenesis-2.6.jar Download https://jcenter.bintray.com/com/fasterxml/jackson/core/jackson-core/2.9.5/jackson-core-2.9.5.jar Download https://jcenter.bintray.com/org/apache/commons/commons-compress/1.16/commons-compress-1.16.jar Download https://jcenter.bintray.com/com/github/zafarkhaja/java-semver/0.9.0/java-semver-0.9.0.jar > Task :buildSrc:jar :jar: No valid plugin descriptors were found in META-INF/gradle-plugins Download https://plugins.gradle.org/m2/gradle/plugin/net/vivin/gradle-semantic-build-versioning/4.0.0/gradle-semantic-build-versioning-4.0.0.pom Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.8.0.201706111038-r/org.eclipse.jgit-4.8.0.201706111038-r.pom Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit-parent/4.8.0.201706111038-r/org.eclipse.jgit-parent-4.8.0.201706111038-r.pom Download https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.pom Download https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom Download https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.pom Download https://plugins.gradle.org/m2/org/slf4j/slf4j-parent/1.7.2/slf4j-parent-1.7.2.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcomponents-client/4.3.6/httpcomponents-client-4.3.6.pom Download https://plugins.gradle.org/m2/commons-codec/commons-codec/1.6/commons-codec-1.6.pom Download https://plugins.gradle.org/m2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.pom Download https://plugins.gradle.org/m2/org/apache/commons/commons-parent/28/commons-parent-28.pom Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcomponents-core/4.3.3/httpcomponents-core-4.3.3.pom Download https://plugins.gradle.org/m2/org/apache/commons/commons-parent/22/commons-parent-22.pom Download https://plugins.gradle.org/m2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar Download https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar Download https://plugins.gradle.org/m2/gradle/plugin/net/vivin/gradle-semantic-build-versioning/4.0.0/gradle-semantic-build-versioning-4.0.0.jar Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar Download https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar Download https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.jar Download https://plugins.gradle.org/m2/commons-codec/commons-codec/1.6/commons-codec-1.6.jar Download https://plugins.gradle.org/m2/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar Download https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.8.0.201706111038-r/org.eclipse.jgit-4.8.0.201706111038-r.jar Download https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.pom Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.pom Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/39/commons-parent-39.pom Download https://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom Download https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar Download https://repo.maven.apache.org/maven2/junit/junit/4.12/junit-4.12.jar Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar > Task :cruise-control-core:compileJava Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control-core/src/main/java/com/linkedin/cruisecontrol/common/config/ConfigDef.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2.pom Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.pom Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.10/zookeeper-3.4.10.pom Download https://repo.maven.apache.org/maven2/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.pom Download https://repo.maven.apache.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.pom Download https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.pom Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2.jar Download https://repo.maven.apache.org/maven2/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.jar Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.jar Download https://repo.maven.apache.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar Download https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.10/zookeeper-3.4.10.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-server/9.4.6.v20170531/jetty-server-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.pom Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.2.2/metrics-core-3.2.2.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlet/9.4.6.v20170531/jetty-servlet-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-project/9.4.6.v20170531/jetty-project-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-parent/3.2.2/metrics-parent-3.2.2.pom Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom Download https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom Download https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom Download https://repo.maven.apache.org/maven2/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.6.v20170531/jetty-io-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/9.4.6.v20170531/jetty-http-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/net/java/jvnet-parent/3/jvnet-parent-3.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.6.v20170531/jetty-security-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.pom Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.6.v20170531/jetty-util-9.4.6.v20170531.pom Download https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-security/9.4.6.v20170531/jetty-security-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-servlet/9.4.6.v20170531/jetty-servlet-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-server/9.4.6.v20170531/jetty-server-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/9.4.6.v20170531/jetty-http-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-core/3.2.2/metrics-core-3.2.2.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/9.4.6.v20170531/jetty-io-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/9.4.6.v20170531/jetty-util-9.4.6.v20170531.jar Download https://repo.maven.apache.org/maven2/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.11.11/scala-compiler-2.11.11.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.11.11/scala-reflect-2.11.11.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.11.11/scala-reflect-2.11.11.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.11.11/scala-compiler-2.11.11.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.jar Download https://repo.maven.apache.org/maven2/com/typesafe/zinc/zinc/0.3.15/zinc-0.3.15.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/compiler-interface/0.13.15/compiler-interface-0.13.15.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/incremental-compiler/0.13.15/incremental-compiler-0.13.15.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/sbt-interface/0.13.15/sbt-interface-0.13.15.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.10.6/scala-compiler-2.10.6.pom Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.pom Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/incremental-compiler/0.13.15/incremental-compiler-0.13.15.jar Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/sbt-interface/0.13.15/sbt-interface-0.13.15.jar Download https://repo.maven.apache.org/maven2/com/typesafe/zinc/zinc/0.3.15/zinc-0.3.15.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.6/scala-library-2.10.6.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.jar Download https://repo.maven.apache.org/maven2/org/scala-lang/scala-compiler/2.10.6/scala-compiler-2.10.6.jar Download https://repo.maven.apache.org/maven2/com/typesafe/sbt/compiler-interface/0.13.15/compiler-interface-0.13.15-sources.jar > Task :cruise-control:compileScala Pruning sources from previous analysis, due to incompatible CompileSetup. ………
在构建的过程中，你可能会遇到如下的报错信息：
如果遇到上面的报错信息时，不要慌，根据报错信息提示缺少javac这个包，我们可以自行去oracle官网下载相应的jdk源码包，将对应的javac包拷贝到指定目录中即可。我服务器端已经安装好了对应的jdk源码包，于是我做了如下操作：
[root@yinzhengjie cruise-control]# find / -name javac /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac [root@yinzhengjie cruise-control]# cp /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/bin [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# find / -name javac /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre/bin/javac /data/yinzhengjie/softwares/jdk/jdk1.8.0_131/bin/javac [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew clean jar To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing > Task :cruise-control-core:compileJava Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control-core/src/main/java/com/linkedin/cruisecontrol/common/config/ConfigDef.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :cruise-control:compileScala Pruning sources from previous analysis, due to incompatible CompileSetup. Note: /data/yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/analyzer/AnalyzerUtils.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.pom Download https://repo.maven.apache.org/maven2/org/easymock/easymock/3.4/easymock-3.4.pom Download https://repo.maven.apache.org/maven2/org/easymock/easymock-parent/3.4/easymock-parent-3.4.pom Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12.pom Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.2/objenesis-2.2.pom Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis-parent/2.2/objenesis-parent-2.2.pom Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.jar Download https://repo.maven.apache.org/maven2/org/easymock/easymock/3.4/easymock-3.4.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2-test.jar Download https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.11.0.2/kafka_2.11-0.11.0.2-test.jar Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12.jar Download https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.2/objenesis-2.2.jar Download https://jcenter.bintray.com/com/linkedin/kafka/clients/li-apache-kafka-clients/0.0.12/li-apache-kafka-clients-0.0.12-tests.jar Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 38s 10 actionable tasks: 10 executed [root@yinzhengjie cruise-control]#
3>.修改broker配置文件
构建成功后，我们可以在项目目录下的cruise-control-metrics-reporter/build/libs下会生成一个jar包，我们将这个jar包拷贝到broker安装目录的libs目录下，具体操作如下：
[root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ll cruise-control-metrics-reporter/build/libs/ total 36 -rw-r--r-- 1 root root 36547 Dec 4 11:00 cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# scp cruise-control-metrics-reporter/build/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 10.1.2.118:/soft/kafka/libs/ root@10.1.2.118's password: cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 100% 36KB 35.7KB/s 00:00 [root@yinzhengjie cruise-control]#
注意，上面的操作需要在每台broker都需要做同样的操作，之后需要在每台broker端的配置文件“server.properties”中新增下列行，保存之后启动broker：
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
4>.修改cruisecontrol项目目录下的conf/cruisecontrol.properties文件（根据你实际生产环境自定义相应的参数即可）
[root@yinzhengjie cruise-control]# pwd /yinzhengjie/softwares/kafkasoft/kafka-cruiseControl/cruise-control [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 10:45 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep bootstrap.servers bootstrap.servers=10.1.2.118:9092,10.1.2.117:9092,10.1.2.116:9092,10.1.2.115:9092,10.1.2.114:9092 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep zookeeper.connect zookeeper.connect=10.1.2.114:2181,10.1.2.115:2181,10.1.2.116:2181,10.1.2.117:2181,10.1.2.118:2181/kafka_yinzhengjie_cluster1 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep webserver.http.port webserver.http.port=7070 [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# cat config/cruisecontrol.properties | grep -v ^# | grep webserver.http.cors.origin webserver.http.cors.origin=http://localhost:7070/ [root@yinzhengjie cruise-control]#
[root@node118 ~]# cat `which xcall.sh ` #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -lt 1 ];then echo "请输入参数" exit fi #获取用户输入的命令 cmd=$@ for (( i=114;i<=118;i++ )) do #使终端变绿色 tput setaf 2 echo ============= node${i}.aggrx : $cmd ============ #使终端变回原来的颜色，即白灰色 tput setaf 7 #远程执行命令 ssh node${i}.aggrx $cmd #判断命令是否执行成功 if [ $? == 0 ];then echo "命令执行成功" fi done [root@node118 ~]#
[root@node118 ~]# cat `which xkafka.sh ` #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exit fi #获取用户输入的命令 cmd=$1 for (( i=114 ; i<=118 ; i++ )) ; do tput setaf 2 echo ========== node${i}.aggrx $cmd ================ tput setaf 9 case $cmd in start) ssh node${i}.aggrx "source /etc/profile ; kafka-server-start.sh -daemon /soft/kafka/config/server.properties" echo node${i}.aggrx "服务已启动" ;; stop) ssh node${i}.aggrx "source /etc/profile ; kafka-server-stop.sh" echo node${i}.aggrx "服务已停止" ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;; esac done [root@node118 ~]#
[root@node118 ~]# xcall.sh jps ============= node114.aggrx : jps ============ 29930 QuorumPeerMain 1900 Jps 命令执行成功 ============= node115.aggrx : jps ============ 788 QuorumPeerMain 5023 Jps 命令执行成功 ============= node116.aggrx : jps ============ 19335 QuorumPeerMain 23516 Jps 命令执行成功 ============= node117.aggrx : jps ============ 26770 Jps 22639 QuorumPeerMain 命令执行成功 ============= node118.aggrx : jps ============ 20017 QuorumPeerMain 24949 Jps 30056 Bootstrap 命令执行成功 [root@node118 ~]# [root@node118 ~]# [root@node118 ~]# xkafka.sh start ========== node114.aggrx start ================ node114.aggrx 服务已启动 ========== node115.aggrx start ================ node115.aggrx 服务已启动 ========== node116.aggrx start ================ node116.aggrx 服务已启动 ========== node117.aggrx start ================ node117.aggrx 服务已启动 ========== node118.aggrx start ================ node118.aggrx 服务已启动 [root@node118 ~]# [root@node118 ~]# [root@node118 ~]# xcall.sh jps ============= node114.aggrx : jps ============ 2256 Kafka 2480 Jps 29930 QuorumPeerMain 命令执行成功 ============= node115.aggrx : jps ============ 788 QuorumPeerMain 5588 Jps 5372 Kafka 命令执行成功 ============= node116.aggrx : jps ============ 19335 QuorumPeerMain 23865 Kafka 24079 Jps 命令执行成功 ============= node117.aggrx : jps ============ 27173 Kafka 27387 Jps 22639 QuorumPeerMain 命令执行成功 ============= node118.aggrx : jps ============ 20017 QuorumPeerMain 25556 Jps 25317 Kafka 30056 Bootstrap 命令执行成功 [root@node118 ~]#
[root@node118 ~]# xcall.sh lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============= node114.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 2256 root mem REG 253,0 36547 100826940 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 2256 root 13r REG 253,0 36547 100826940 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令执行成功 ============= node115.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 5372 root mem REG 253,0 36547 67503111 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 5372 root 13r REG 253,0 36547 67503111 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令执行成功 ============= node116.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 23865 root mem REG 253,0 36547 1046 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 23865 root 13r REG 253,0 36547 1046 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令执行成功 ============= node117.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 27173 root mem REG 253,0 36547 83946551 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 27173 root 13r REG 253,0 36547 83946551 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令执行成功 ============= node118.aggrx : lsof /soft/kafka/libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar ============ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 25317 root mem REG 253,0 36547 59044906 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar java 25317 root 13r REG 253,0 36547 59044906 /soft/kafka_2.11-1.1.0//libs/cruise-control-metrics-reporter-0.1.18-SNAPSHOT.jar 命令执行成功 [root@node118 ~]#
5>.启动cruisecontrol服务
[root@yinzhengjie cruise-control]# ll total 68 -rw-r--r-- 1 root root 8257 Dec 4 10:45 build.gradle drwxr-xr-x 5 root root 61 Dec 4 10:47 buildSrc drwxr-xr-x 2 root root 50 Dec 4 10:45 checkstyle drwxr-xr-x 2 root root 80 Dec 4 11:44 config -rw-r--r-- 1 root root 1079 Dec 4 10:45 CONTRIBUTING.md drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-core drwxr-xr-x 4 root root 28 Dec 4 11:00 cruise-control-metrics-reporter drwxr-xr-x 3 root root 19 Dec 4 10:45 docs drwxr-xr-x 3 root root 47 Dec 4 10:45 gradle -rw-r--r-- 1 root root 86 Dec 4 10:45 gradle.properties -rwxr-xr-x 1 root root 5299 Dec 4 10:45 gradlew -rw-r--r-- 1 root root 2260 Dec 4 10:45 gradlew.bat -rwxr-xr-x 1 root root 5129 Dec 4 10:45 kafka-cruise-control-start.sh -rw-r--r-- 1 root root 1332 Dec 4 10:45 LICENSE -rw-r--r-- 1 root root 304 Dec 4 10:45 NOTICE -rw-r--r-- 1 root root 11476 Dec 4 10:45 README.md -rw-r--r-- 1 root root 153 Dec 4 10:45 semantic-build-versioning.gradle -rw-r--r-- 1 root root 911 Dec 4 10:45 settings.gradle [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]# ./gradlew jar copyDependantLibs To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.8/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.54/bcpkix-jdk15on-1.54.pom Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.54/bcprov-jdk15on-1.54.pom Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.54/bcpkix-jdk15on-1.54.jar Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.54/bcprov-jdk15on-1.54.jar Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1m 59s 11 actionable tasks: 4 executed, 7 up-to-date [root@yinzhengjie cruise-control]#
[root@yinzhengjie cruise-control]# ./kafka-cruise-control-start.sh config/cruisecontrol.properties [2018-12-04 12:27:10,878] INFO KafkaCruiseControlConfig values: anomaly.detection.allow.capacity.estimation = true anomaly.detection.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal] anomaly.detection.interval.ms = 10000 anomaly.notifier.class = class com.linkedin.kafka.cruisecontrol.detector.notifier.SelfHealingNotifier bootstrap.servers = [10.1.2.118:9092, 10.1.2.117:9092, 10.1.2.116:9092, 10.1.2.115:9092, 10.1.2.114:9092] broker.capacity.config.resolver.class = class com.linkedin.kafka.cruisecontrol.config.BrokerCapacityConfigFileResolver broker.metric.sample.aggregator.completeness.cache.size = 5 broker.metrics.window.ms = 300000 client.id = kafka-cruise-control completed.user.task.retention.time.ms = 21600000 connections.max.idle.ms = 540000 cpu.balance.threshold = 1.1 cpu.capacity.threshold = 0.8 cpu.low.utilization.threshold = 0.0 default.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal] disk.balance.threshold = 1.1 disk.capacity.threshold = 0.8 disk.low.utilization.threshold = 0.0 execution.progress.check.interval.ms = 10000 failed.brokers.zk.path = /CruiseControlBrokerList follower.network.inbound.weight.for.cpu.util = 0.3 goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerDiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerEvenRackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PreferredLeaderElectionGoal] hard.goals = [com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal] leader.network.inbound.weight.for.cpu.util = 0.6 leader.network.outbound.weight.for.cpu.util = 0.1 linear.regression.model.cpu.util.bucket.size = 5 linear.regression.model.min.num.cpu.util.buckets = 5 linear.regression.model.required.samples.per.bucket = 100 max.active.user.tasks = 5 max.allowed.extrapolations.per.broker = 5 max.allowed.extrapolations.per.partition = 5 max.cached.completed.user.tasks = 100 max.replicas.per.broker = 10000 metadata.max.age.ms = 55000 metric.anomaly.finder.class = [com.linkedin.kafka.cruisecontrol.detector.KafkaMetricAnomalyFinder] metric.sampler.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler metric.sampler.partition.assignor.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.DefaultMetricSamplerPartitionAssignor metric.sampling.interval.ms = 120000 min.samples.per.broker.metrics.window = 1 min.samples.per.partition.metrics.window = 1 min.valid.partition.ratio = 0.995 network.inbound.balance.threshold = 1.1 network.inbound.capacity.threshold = 0.8 network.inbound.low.utilization.threshold = 0.0 network.outbound.balance.threshold = 1.1 network.outbound.capacity.threshold = 0.8 network.outbound.low.utilization.threshold = 0.0 num.broker.metrics.windows = 20 num.cached.recent.anomaly.states = 10 num.concurrent.leader.movements = 1000 num.concurrent.partition.movements.per.broker = 10 num.metric.fetchers = 1 num.partition.metrics.windows = 1 num.proposal.precompute.threads = 1 partition.metric.sample.aggregator.completeness.cache.size = 5 partition.metrics.window.ms = 300000 proposal.expiration.ms = 60000 receive.buffer.bytes = 32768 reconnect.backoff.ms = 50 replica.count.balance.threshold = 1.1 request.timeout.ms = 30000 sample.store.class = class com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS topics.excluded.from.partition.movement = use.linear.regression.model = false webserver.accesslog.enabled = true webserver.accesslog.path = access.log webserver.accesslog.retention.days = 14 webserver.api.urlprefix = /kafkacruisecontrol/* webserver.http.address = 0.0.0.0 webserver.http.cors.allowmethods = OPTIONS,GET,POST webserver.http.cors.enabled = false webserver.http.cors.exposeheaders = User-Task-ID webserver.http.cors.origin = http://localhost:7070/ webserver.http.port = 7070 webserver.request.maxBlockTimeMs = 10000 webserver.session.maxExpiryTimeMs = 60000 webserver.session.path = / webserver.ui.diskpath = ./cruise-control-ui/dist/ webserver.ui.urlprefix = /* zookeeper.connect = 10.1.2.114:2181,10.1.2.115:2181,10.1.2.116:2181,10.1.2.117:2181,10.1.2.118:2181/kafka_yinzhengjie_cluster1/ (com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig) [2018-12-04 12:27:11,102] INFO ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [10.1.2.118:9092, 10.1.2.117:9092, 10.1.2.116:9092, 10.1.2.115:9092, 10.1.2.114:9092] check.crcs = true client.id = CruiseControlMetricsReporterSampler-9001457965431329813-consumer--1607149276 connections.max.idle.ms = 540000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = CruiseControlMetricsReporterSampler-9001457965431329813 heartbeat.interval.ms = 3000 interceptor.classes = null internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 2147483647 max.poll.records = 2147483647 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 305000 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'anomaly.detection.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'network.outbound.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'disk.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'metric.anomaly.percentile.lower.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'replica.count.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'min.monitored.partition.percentage' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'cpu.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'num.partition.metrics.windows' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'execution.progress.check.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'capacity.config.file' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,152] WARN The configuration 'webserver.http.cors.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'partition.metric.sample.store.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'network.outbound.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'cpu.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.ui.urlprefix' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'proposal.expiration.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.api.urlprefix' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.sampler.partition.assignor.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'webserver.http.cors.origin' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.sampler.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'hard.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'self.healing.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'broker.metrics.window.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'cpu.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,153] WARN The configuration 'metric.anomaly.finder.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'max.cached.completed.user.tasks' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'num.sample.loading.threads' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'zookeeper.connect' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'disk.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.http.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'partition.metrics.window.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'disk.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'network.inbound.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'network.inbound.capacity.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.session.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'failed.brokers.zk.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.accesslog.enabled' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'num.concurrent.partition.movements.per.broker' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,154] WARN The configuration 'webserver.ui.diskpath' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'completed.user.task.retention.time.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.reporter.topic.pattern' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'min.samples.per.broker.metrics.window' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'num.broker.metrics.windows' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.accesslog.retention.days' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.sampling.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'min.samples.per.partition.metrics.window' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'anomaly.notifier.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.http.cors.exposeheaders' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'anomaly.detection.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'network.outbound.low.utilization.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'num.proposal.precompute.threads' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.http.address' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'metric.anomaly.analyzer.metrics' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'network.inbound.balance.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,155] WARN The configuration 'webserver.session.maxExpiryTimeMs' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.accesslog.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.request.maxBlockTimeMs' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'metric.anomaly.percentile.upper.threshold' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'sample.store.topic.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'max.replicas.per.broker' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'broker.metric.sample.store.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'webserver.http.cors.allowmethods' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'num.metric.fetchers' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'max.active.user.tasks' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'sample.store.class' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,156] WARN The configuration 'default.goals' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2018-12-04 12:27:11,159] INFO Kafka version : 0.11.0.2 (org.apache.kafka.common.utils.AppInfoParser) [2018-12-04 12:27:11,159] INFO Kafka commitId : 73be1e1168f91ee2 (org.apache.kafka.common.utils.AppInfoParser) Exception in thread "main" java.lang.IllegalStateException: Cruise Control cannot find sampling topic matches __CruiseControlMetrics in the target cluster. at com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler.configure(CruiseControlMetricsReporterSampler.java:203) at com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig.getConfiguredInstance(KafkaCruiseControlConfig.java:1065) at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager.<init>(MetricFetcherManager.java:129) at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager.<init>(MetricFetcherManager.java:69) at com.linkedin.kafka.cruisecontrol.monitor.task.LoadMonitorTaskRunner.<init>(LoadMonitorTaskRunner.java:72) at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.<init>(LoadMonitor.java:152) at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.<init>(LoadMonitor.java:110) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.<init>(KafkaCruiseControl.java:89) at com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.<init>(AsyncKafkaCruiseControl.java:64) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:71) [root@yinzhengjie cruise-control]# [root@yinzhengjie cruise-control]#
启动的时候报错了，这就有点小尴尬了，问题还在排除中......
四.Kafka Manager开源监控
我接触的第一款kafka监控工具，配置简单，大师不支持告警，详情请参考：https://www.cnblogs.com/yinzhengjie/p/9808121.html。
五.Kafka Eagle
我接触的第二款Kafka监控工具，配置也简单，想必Kafka Manager更优秀一点，支持告警功能，详情请参考：https://www.cnblogs.com/yinzhengjie/p/9957389.html。
六.JmxTool
严格的来说，JmxTools不是一个框架，而是kafka社区默认提供的工具，用于实时查看JMX监控指标。如果用户一时找不到合适的工具来监测JMX指标，可以考虑使用“临时救急”。打开一个终端并进入Kafka安装目录bin下，输入命令“kafka-run-class.sh kafka.tools.JmxTool”便可以得到JmxTool工具的帮助信息：
[root@yinzhengjie cruise-control]# kafka-run-class.sh kafka.tools.JmxTool Dump JMX values to standard output. Option Description ------ ----------- --attributes <String: name> #指定 CSV格式的属性，如果不指定该参数则默认读取所有属性。查询属性的白名单。这个是逗号分隔的列表。如果没有属性指定所有对象将被询问。 --date-format <String: format> # 设置时间格式，比如设置为 YYYY-MM-ss,则输出时间戳即为2018-12-01这样的格式 --help #查询帮助信息 --jmx-url <String: service-url> #指定要链接的JMX接口，默认格式是service.jmx.rmi:///jndi/rmi://:<JMX端口>/jmxrmi --object-name <String: name> #指定JMX的Mbean名称 --reporting-interval <Integer: ms> #设置实时监测时间间隔，默认每2秒打印一次JMX指标值。 [root@yinzhengjie cruise-control]#
假设我们要实时监控JMX MBean的消息入站速率。我们想要每个3秒输出一次该MBean过去15分钟的平均值，则可以执行下面的命令：
[root@yinzhengjie cruise-control]# kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec --jmx-url service:jmx:rmi:///jndi/rmi://10.1.3.116:12345/jmxrmi --date-format "YYYY-MM-dd HH:mm:ss" --attributes FifteenMinuteRate --reporting-interval 3000 Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://10.1.3.116:12345/jmxrmi. "time","kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:FifteenMinuteRate" 2018-12-04 13:21:25,1.0616180583446175E7 2018-12-04 13:21:28,1.0610205773377854E7 2018-12-04 13:21:31,1.0610205773377854E7 2018-12-04 13:21:34,1.0605810471779421E7 2018-12-04 13:21:37,1.0599882204038633E7 2018-12-04 13:21:40,1.0599882204038633E7 2018-12-04 13:21:43,1.059681434484897E7 2018-12-04 13:21:46,1.059681434484897E7 2018-12-04 13:21:49,1.0591635691401184E7 2018-12-04 13:21:52,1.058678141082473E7 2018-12-04 13:21:55,1.058678141082473E7 2018-12-04 13:21:58,1.0581291456978852E7 2018-12-04 13:22:01,1.0581291456978852E7