Kafka集群的搭建
node1 node2 node3
kafka须用版本(kafka-0.8.2.2),否则不兼容spark1.6
1.启动zookeeper集群node1 node2 node3
zkServer.sh start
zkServer.sh status
2.修改kafka中的配置文件server.properties(broker.id 和 zookeeper.connect)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193541844-446381222.png)
broker.id =0 || broker.id=1
port =9092
log.dirs =/tmp/kafka-logs 并不是存放的kafka日志,而是实际的数据
leader的均衡机制,默认为true
zookeeper.connect = node1:2181,node2:2181,node3:2181
node1
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193543828-317530365.png)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193544547-1746253296.png)
node2
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193548203-1301811847.png)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193552063-240454352.jpg)
node3
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193558719-769130077.png)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193559328-389394577.png)
3.启动kafka集群--测试kafka可用性
第一步:
因为kafka没有主从关系,所以集群中的每一台都需要启动Kafka
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193559703-1775065695.png)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193600219-1116377312.png)
nohup /home/kafka_2.10-0.8.2.2/bin/kafka-server-start.sh /home/kafka_2.10-0.8.2.2/config/server.properties &
node1,node2,node3启动命令(启动时编写的脚本中需要指定server.properties)
chmod +x start-kafka.sh
./start-kafka.sh
第二步:
创建topic
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193601516-212191765.png)
/home/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181--replication-factor 3--partitions 3--topic kfk
chmod +x create-topic.sh
./create-topic.sh
第三步:
node1查看刚刚创建的topic
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193601938-1830192051.png)
/home/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181
chmod +x list-topic.sh
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193602297-756245015.png)
同样在node2,node3中均可以看到在node1上创建的topic
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193602891-1501223624.png)
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193603125-1884902727.png)
第四步:
在node1中创建生产者,在node2中创建消费者,node1中生产,查看node2中是否能够消费
node1
vi producer-topic.sh
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193604766-693604277.png)
/home/kafka_2.10-0.8.2.2/bin/kafka-console-producer.sh --broker-list node1:9092,node2:9092,node3:9092--topic kfk
chmod +x producer-topic.sh
./producer-topic.sh
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193606203-1093905296.png)
node2
vi consumer-topic.sh
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193606656-1329688107.png)
/home/kafka_2.10-0.8.2.2/bin/kafka-console-consumer.sh --zookeeper node1:2181,node2:2181,node3:2181--from-beginning --topic kfk
chmod +x consumer-topic.sh
./consumer-topic.sh
![](https://images2015.cnblogs.com/blog/1008304/201703/1008304-20170306193607266-1500758537.png)