• linux上kafka搭建小结


    1、安装java

    yum -y list java

    可以查看java的版本,然后选1.8的64位那个安装

    yum install java_1.8.0_openjdk.x86_64

    或者从网站下载安装

    tar -zxvf jre-8u271-linux-x64.tar.gz
    mkdir /usr/lib/jvm
    mv jre-8u271-linux-x64 /usr/lib/jvm
    chown root:root -R /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64

    java安装之后就开始设置环境变量

    vi /etc/profile

    #set java environment
    JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64
    PATH=$PATH:$JAVA_HOME/bin
    CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export JAVA_HOME CLASSPATH PATH

    使之生效命令

    source /etc/profile

    或者

    .  /etc/profile

    2、 zookeeper

    装kafka的时候才知道,必须要先安装zookeeper

    从官网下载了3.5.5的包

    http://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/

    tar zxvf apache-zookeeper-3.5.5-bin.tar.gz
    cd apache-zookeeper-3.5.5-bin
    cd conf
    cp zoo_sample.cfg zoo.cfg
    vi zoo.cfg

    修改dataDir=后面的路径,就改成程序路径下,然后mkdir data  | cd data | mkdir zookeeper | mkdir log

    修改环境变量/etc/profile

    ZOOKEEPER_HOME=/root/hzt/apache-zookeeper-3.5.5-bin
    PATH=$PATH:$ZOOKEEPER_HOME/bin

    并生效

    source /etc/profile

    启动程序

    sh zkServer.sh start

    停止命令把start改成stop就行。

    测试下可以用windows的cmd

    telnet 替换实际的机器ip 2181

    或者linux下看状态

    sh zkServer.sh status

    查看状态发现了一个状况

    ZooKeeper JMX enabled by default
    Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
    Client port found: 2181. Client address: localhost.
    Error contacting service. It is probably not running.

    查看logs文件夹下的.out文件

    2020-12-14 16:59:13,825 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0
    .0:2181
    2020-12-14 16:59:13,825 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
    2020-12-14 16:59:13,830 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set 
    to 3
    2020-12-14 16:59:13,832 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to
     0
    2020-12-14 16:59:13,832 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
    2020-12-14 16:59:13,832 [myid:] - WARN  [main:QuorumPeerMain@125] - Either no config or no quorum define
    d

    zoo.cfg中datadir也配置过了,修改了下也没效果。然后把系统中开启的tomcat关掉后再启动就行了。不知道是不是跟8080端口被占用有关系

    [opt@test bin]$ zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
    Client port found: 2181. Client address: localhost.
    Mode: standalone

    3、安装kafka

    从官网下载非源码版本的程序,带有src安装了会有问题

    tar xzvf kafka_2.12-2.3.0.tgz

    cd kafka_2.12-2.3.0

    启动程序

    bin/kafka-server-start.sh config/server.properties &

    检测2181与9092端口

    netstat -tunlp|egrep '(2181|9092)'

    在用普通用户启动时候会报

    Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
    [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
    [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
    ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
    换成root用户启动就好了,怀疑普通用户没有默认配置中跟路径权限。后期把配置修改下再试试。
    功能测试,新开一个窗口作为生产者

    运行producer
    bin/kafka-console-producer.sh --broker-list 192.168.1.1:9092 --topic test
    再开一个窗口作为消费者
    sh kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.1:9092 --topic test --from-beginning
    然后生产者界面输入的内容会在消费者界面出现
    4、客户端开发
    从github上download相应的src,使用librdkafka
    unzip librdkafka-master.zip
    cd librdkafka-master
    sh ./configure
    make
    sudo make install

    https://github.com/edenhill/librdkafka
    完了之后可以进examples文件夹中运行demo
    生产者:
    ./producer kafka机器的ip:9092 topic
    然后就可以丢数据进去了
    消费者:
    ./consumer
    kafka机器的ip:9092 groupid(Consumer group id) topics
    可以订阅一个或者多个主题,组id的用途还没搞懂

    如果遇到连接被kafka拒绝的情况,则需要修改下kafka的conf/server.properties中的listeners这项

    本机运行的话可以改成
    listeners=PLAINTEXT://localhost:9092

    参考文档:
    1、http://www.xwood.net/_site_domain_/_root/5870/5930/5932/t_c261426.html
    2、https://www.cnblogs.com/wjup/p/11041274.html
    3、https://blog.csdn.net/qq_41094332/article/details/104366315
    4、https://www.cnblogs.com/justuntil/p/8033792.html
    5、https://www.cnblogs.com/wangyangliuping/p/5546506.html
  • 相关阅读:
    14.7.1 重新调整InnoDB系统表空间的大小
    css代码引入方式与基本选择器
    html常用标签
    python数据库操作
    pt-osc使用一则
    mysql优化一则(20170703)
    进程与生产者消费者模型
    面向对象程序设计
    常用模块
    模块与包
  • 原文地址:https://www.cnblogs.com/ikel/p/14053465.html
Copyright © 2020-2023  润新知