kafka 集群安装

环境：

10.23.209.70 zookeep-kafka-node1  zookeep+kafka  CentOS Linux release 7.1.1503 
10.23.209.71 zookeep-kafka-node2  zookeep+kafka  CentOS Linux release 7.1.1503
10.23.209.72 zookeep-kafka-node3  zookeep+kafka  CentOS Linux release 7.1.1503

官方网站：

JDK下载地址：https://www.oracle.com/technetwork/java/javase/downloads/index.html
Zookeep官方下载地址：https://zookeeper.apache.org/releases.html
kafka官方下载地址：http://kafka.apache.org/downloads

前期准备关闭防火墙，selinux

[root@zookeep-kafka-node1 ~]# setenforce 0
[root@zookeep-kafka-node1 ~]# getenforce
Permissive
[root@zookeep-kafka-node1 ~]# cat /etc/selinux/config | grep 'SELINUX=' && -v ^#
# SELINUX= can take one of these three values:
SELINUX=disabled
[root@zookeep-kafka-node1 ~]# systemctl stop firewalld
[root@zookeep-kafka-node1 ~]# systemctl disable firewalld

一.JDK1.8安装

1.解压jdk至指定目录

[root@zookeep-kafka-node1 local]# tar zxvf jdk-8u231-linux-x64.tar.gz  -C /usr/local/

2.修改vi /etc/profile 增加JDK环境变量

export JAVA_HOME=/usr/local/jdk1.8.0_231
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

3.使环境变量生效

[root@zookeep-kafka-node1 local]# source /etc/profile

4.测试java环境，查看到java信息说明部署成功

[root@zookeep-kafka-node1 local]# java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

5.将JDK，SCP到其他2台机器上。并修改/etc/profile文件使其生效  JDK部署完毕

[root@zookeep-kafka-node1 local]# scp -r /usr/local/jdk1.8.0_231/  root@10.23.215.71:/usr/local/
[root@zookeep-kafka-node1 local]# scp -r /usr/local/jdk1.8.0_231/  root@10.23.215.72:/usr/local/

二.部署zookeep集群

1.解压zookeep压缩包至指定文件夹

tar zxvf apache-zookeeper-3.5.6-bin.tar.gz -C /opt/
mv /opt/apache-zookeeper-3.5.6-bin/ zookeeper-3.5.6

2.创建zookeep配置文件zoo.cfg内容

[root@zookeep-kafka-node1 conf]# cat zoo_sample.cfg | grep -v ^$|grep -v ^# >zoo.cfg
[root@zookeep-kafka-node1 conf]# cat zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181

修改配置把数据目录和日志目录分开，并增加集群信息

[root@zookeep-kafka-node1 conf]# cat zoo.cfg
tickTime=10000
initLimit=5
syncLimit=2
dataDir /opt/zookeeper-3.5.6/data/zkdata
dataLogDir=/opt/zookeeper-3.5.6/data/zklogs
clientPort=2181
server.1=10.23.209.70:2888:3888
server.2=10.23.209.71:2888:3888
server.3=10.23.209.72:2888:3888


创建数据和日志目录
[root@zookeep-kafka-node1 opt]# mkdir -p /opt/zookeeper-3.5.6/data/{zkdata,zklogs}

将zookeep scp到其他2台服务器
[root@zookeep-kafka-node1 opt]# scp -r /opt/zookeeper-3.5.6 root@10.23.215.71:/opt
[root@zookeep-kafka-node1 opt]# scp -r /opt/zookeeper-3.5.6 root@10.23.215.72:/opt

3.创建myid文件 

[root@zookeep-kafka-node1 opt]# echo "1" > /opt/zookeeper-3.5.6/data/zkdata/myid

[root@zookeep-kafka-node2 opt]# echo "2" > /opt/zookeeper-3.5.6/data/zkdata/myid

[root@zookeep-kafka-node3 opt]# echo "3" > /opt/zookeeper-3.5.6/data/zkdata/myid

4.启动zookeep服务 看到2个follower 1个leader 集群正常

[root@zookeep-kafka-node1 bin]# /opt/zookeeper-3.5.6/bin/zkServer.sh  start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeep-kafka-node1 bin]# /opt/zookeeper-3.5.6/bin/zkServer.sh  status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.5.6/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

三.安装kafka

1.解压至指定目录，修改配置

[root@zookeep-kafka-node1 ~]# tar zxvf kafka_2.12-2.3.0.tgz  -C /opt/
[root@zookeep-kafka-node1 ~]# cd /opt/kafka_2.12-2.3.0/config/
[root@zookeep-kafka-node1 config]# cp server.properties  server.properties.bak
[root@zookeep-kafka-node1 config]# grep -v ^# server.properties.bak | grep -v ^$ >server.properties
[root@zookeep-kafka-node1 config]# cat server.properties
broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

修改数据存放路径 集群信息 等
[root@zookeep-kafka-node1 config]# cat server.properties 
broker.id=70
port=9092
host.name=10.23.209.70
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka_2.12-2.3.0/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181
zookeeper.connection.timeout.ms=1000000
zookeeper.session.timeout.ms=1000000
delete.topic.enable=true

2.将kafka目录scp到其他2台服务器 并修改文件中的broker.id host.name

[root@zookeep-kafka-node2 config]# cat server.properties
broker.id=71
port=9092
host.name=10.23.209.71
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka_2.11-2.0.0/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181
zookeeper.connection.timeout.ms=1000000
zookeeper.session.timeout.ms=1000000
delete.topic.enable=true

[root@zookeep-kafka-node3 config]# cat server.properties
broker.id=72
port=9092
host.name=10.23.209.72
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka_2.11-2.0.0/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=10.23.209.70:2181,10.23.209.71:2181,10.23.209.72:2181
zookeeper.connection.timeout.ms=1000000
zookeeper.session.timeout.ms=1000000
delete.topic.enable=true

启动kafka

[root@zookeep-kafka-node1 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh  /opt/kafka_2.12-2.3.0/config/server.properties

[root@zookeep-kafka-node2 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh  /opt/kafka_2.12-2.3.0/config/server.properties

[root@zookeep-kafka-node3 opt]# /opt/kafka_2.12-2.3.0/bin/kafka-server-start.sh  /opt/kafka_2.12-2.3.0/config/server.properties

[2019-11-20 12:57:07,175] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:07,175] INFO Kafka startTimeMs: 1574225827167 (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:07,200] INFO [KafkaServer id=70] started (kafka.server.KafkaServer)


[2019-11-20 12:57:26,017] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:26,017] INFO Kafka startTimeMs: 1574225846011 (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:26,019] INFO [KafkaServer id=71] started (kafka.server.KafkaServer)


[2019-11-20 12:57:33,535] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:33,535] INFO Kafka startTimeMs: 1574225853528 (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-20 12:57:33,537] INFO [KafkaServer id=72] started (kafka.server.KafkaServer)