Kafka 1.0.0集群安装

环境

主机名	IP 地址	安装路径	系统
sht-sgmhadoopdn-01	172.16.101.58	/opt/kafka_2.12-1.0.0 /opt/kafka（软连接）	CentOS Linux release 7.3.1611 (Core)
sht-sgmhadoopdn-02	172.16.101.59
sht-sgmhadoopdn-03	172.16.101.60

软件下载

Kafka下载

http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz

JDK下载

https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

软件安装

 一. 安装jdk

# wget https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
# tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/

二.设置.bash_profile环境变量

JAVA_HOME=/usr/java/jdk1.8.0_45
KAFKA_HOME=/opt/kafka
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib
PATH=$KAFKA_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HOME/bin
export PATH

三. 安装kafka

# wget http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz -P /opt
# tar -zxf kafka_2.12-1.0.0.tgz
# ln -s kafka_2.12-1.0.0 kafka
# mkdir /opt/kafka/data

软件配置

一. zookeeper配置

1. 文件zookeeper.properties，各节点一致

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/kafka/data
clientPort=2182
server.1=sht-sgmhadoopdn-01:2889:3889
server.2=sht-sgmhadoopdn-02:2889:3889
server.3=sht-sgmhadoopdn-03:2889:3889

 2. 各节点分别创建server-id

sht-sgmhadoopdn-01

# echo 1 > /opt/kafka/data/myid

sht-sgmhadoopdn-02

# echo 2 > /opt/kafka/data/myid

sht-sgmhadoopdn-03

# echo 3 > /opt/kafka/data/myid

3.启动zookeeper

# /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

 4.查看zookeeper集群状态

# echo stat | nc sht-sgmhadoopdn-01 2182 | grep Mode
Mode: follower
# echo stat | nc sht-sgmhadoopdn-02 2182 | grep Mode
Mode: leader
# echo stat | nc sht-sgmhadoopdn-03 2182 | grep Mode
Mode: follower

 二. kafka配置

1.各节点配置文件server.properties

sht-sgmhadoopdn-01

broker.id=0
listeners=PLAINTEXT://172.16.101.58:9092
advertised.listeners=PLAINTEXT://172.16.101.58:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182

sht-sgmhadoopdn-02

broker.id=1
listeners=PLAINTEXT://172.16.101.59:9092
advertised.listeners=PLAINTEXT://172.16.101.59:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182

 sht-sgmhadoopdn-03

broker.id=2
listeners=PLAINTEXT://172.16.101.60:9092
advertised.listeners=PLAINTEXT://172.16.101.60:9092
log.dirs=/opt/kafka/data
zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182

 2. 启动kafka

# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

3. 查看集群状态

# echo dump | nc sht-sgmhadoopdn-01 2182 | grep broker
    /brokers/ids/0
    /brokers/ids/1
    /brokers/ids/2

三 测试

1. 创建topic

# kafka-topics.sh --zookeeper sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182 --create --partitions 3 --replication-factor 3 --topic test-topic
Created topic "test-topic".
# kafka-topics.sh --zookeeper sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182 --describe --topic test-topic
Topic:test-topic    PartitionCount:3    ReplicationFactor:3    Configs:
    Topic: test-topic    Partition: 0    Leader: 1    Replicas: 1,2,0    Isr: 1,2,0
    Topic: test-topic    Partition: 1    Leader: 2    Replicas: 2,0,1    Isr: 2,0,1
    Topic: test-topic    Partition: 2    Leader: 0    Replicas: 0,1,2    Isr: 0,1,2

2. 生产者生产数据

# kafka-console-producer.sh --broker-list sht-sgmhadoopdn-01:9092,sht-sgmhadoopdn-02:9092,sht-sgmhadoopdn-03:9092 --topic test-topic
>Hello
>This is my fisrt Kafka !

3. 消费者消费数据

# kafka-console-consumer.sh --bootstrap-server bootstrap-servers=sht-sgmhadoopdn-01:9092,sht-sgmhadoopdn-02:9092,sht-sgmhadoopdn-03:9092 --topic test-topic --from-beginning
Hello
This is my fisrt Kafka !