• windows 下部署kafka 日记 转


    一、下载
      去apache 的官网(http://kafka.apache.org/downloads.html)下载最新的二进制版的压
      缩包。目前的最新版本是kafka_2.11-0.8.2.1.tgz。
    二、解压
      直接解压到D 盘根目录下。
    三、修改配置文件
      注意版本不同,可能配置文件不同。请参照实际情况修改。
      1.修改log4j.properties 文件中的“kafka.logs.dir=logs ”为“kafka.logs.dir=/tmp/logs”。
      2.修改test-log4j.properties 文件中的4 处“ File=logs/server.log ” 为 “File=/tmp/logs/server.log”。
      3.复制config/server.properties 为 第一个节点server-9093.properties,修改如下内容:
        broker.id=9093
        port=9093
        host.name=localhost
        log.dirs=/tmp/kafka-logs-9093
      4.复制config/server.properties 为 第一个节点server-9094.properties,修改如下内容:
        broker.id=9094
        port=9094
        host.name=localhost
        log.dirs=/tmp/kafka-logs-9094
    四、修改启动文件
      删除所有不能在windows 下运行的*.sh 文件。复制kafka_2.11-0.8.2.1/bin/windows/*.bat 到kafka_2.11-0.8.2.1/bin 目录下。
      1.修改zookeeper-server-start.bat 文件中的“%~dp0../../”为“%~dp0/../”。
      2.修改kafka-server-start.bat 文件中的“%~dp0../../”为“%~dp0/../”。
      3.修改kafka-run-class.bat 文件中的“pushd %~dp0....”为“pushd %~dp0”。 修改kafka-run-class.bat 文件中的“%BASE_DIR%/config”为“%BASE_DIR%/../config”。
      将kafka-run-class.bat 文件中的如下内容:
      rem Classpath addition for kafka-core dependencies
        for %%i in (%BASE_DIR%coreuilddependant-libs-%SCALA_VERSION%*.jar) do (
        call :concat %%i
      )
      rem Classpath addition for kafka-perf dependencies
        for %%i in (%BASE_DIR%perfuilddependant-libs-%SCALA_VERSION%*.jar) do (
        call :concat %%i
      )
      rem Classpath addition for kafka-clients
        for %%i in (%BASE_DIR%clientsuildlibskafka-clients-*.jar) do (
        call :concat %%i
      )
      rem Classpath addition for kafka-examples
        for %%i in (%BASE_DIR%examplesuildlibskafka-examples-*.jar) do (
        call :concat %%i
      )
      rem Classpath addition for contrib/hadoop-consumer
        for %%i in (%BASE_DIR%contribhadoop-consumeruildlibskafka-hadoop-consumer-*.jar)
        do (
        call :concat %%i
      )
      rem Classpath addition for contrib/hadoop-producer
        for %%i in (%BASE_DIR%contribhadoop-produceruildlibskafka-hadoop-producer-*.jar) do
        (
        call :concat %%i
      )
      rem Classpath addition for release
        for %%i in (%BASE_DIR%libs*.jar) do (
        call :concat %%i
      )
      rem Classpath addition for core
        for %%i in (%BASE_DIR%coreuildlibskafka_%SCALA_BINARY_VERSION%*.jar) do (
        call :concat %%i
      )
      修改为:
      rem Classpath addition for release
        for %%i in (%BASE_DIR%..libs*.jar) do (
        call :concat %%i
      )
    五、启动zookeeper
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令: zookeeper-server-start ../config/zookeeper.properties
    六、启动kafka
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令,启动第一个节点:
      kafka-server-start.bat ..configserver-9093.properties
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令,启动第二个节点:
      kafka-server-start.bat ..configserver-9094.properties
    七、创建topic 批处理脚本  执行报错,不影响 。
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令:
      kafka-topics.bat kafka.admin.TopicCommand %*
    八、创建topic
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令(replication-factor 设置为kafka 节点数):
      kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-replicated-topic
    九、查看topic 描述信息
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令:kafka-topics.bat --describe --zookeeper localhost:2181 --topic my-replicated-topic
    十、生产topic 消息
      打开命令提示符窗口,在kafka_2.11-0.8.2.1/bin 目录下执行如下命令: kafka-console-producer.bat --broker-list localhost:9093 --topic my-replicated-topic
      启动成功后,即可在此窗口输入测试内容,回车后即可发送成功。
    十一、消费topic 消息
      打开命令提示符窗口, 在kafka_2.11-0.8.2.1/bin 目录下执行如下命令:
      kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic
      my-replicated-topic
      启动成功后,即可实时在此窗口看到生产的消息。
    十二、未解决的异常信息
      命令提示符还会输入一些异常信息,还未解决。但是目前看,不影响使用。
      1. 启动kafka 时:
        [2015-05-14 17:36:50,027] INFO Initiating client connection, connectString=localhost:2181
        sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@2e716cb7 (org.apache.zo
        okeeper.ZooKeeper)
        [2015-05-14 17:36:50,058] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181.
        Will not attempt to authenticate using SASL (java.lang.SecurityException: 无 法定位登录配置) (org.apache.zookeeper.ClientCnxn)
        [2015-05-14 17:36:50,063] INFO Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
        [2015-05-14 17:36:50,110] INFO Session establishment complete on server
        127.0.0.1/127.0.0.1:2181, sessionid = 0x14d51c7fc560000, negotiated timeout = 6000
        (org.apache.zookeeper.ClientCnxn)
        [2015-05-14 17:36:50,112] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
        [2015-05-14 17:36:50,273] INFO Loading logs. (kafka.log.LogManager)
        [2015-05-14 17:36:50,323] INFO Recovering unflushed segment 0 in log my-replicated-topic-0.(kafka.log.Log)
        [2015-05-14 17:36:50,335] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
        java.lang.NullPointerException
        at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
        at kafka.log.Log.loadSegments(Log.scala:179)
        at kafka.log.Log.<init>(Log.scala:67)
        at
        kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
        at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662)
        [2015-05-14 17:36:50,344] INFO Completed load of log my-replicated-topic-0 with log end offset 1 (kafka.log.Log)
        [2015-05-14 17:36:50,351] INFO Recovering unflushed segment 0 in log my-replicated-topic-1.(kafka.log.Log)
        [2015-05-14 17:36:50,355] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
        java.lang.NullPointerException
        at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
        at kafka.log.Log.loadSegments(Log.scala:179)
        at kafka.log.Log.<init>(Log.scala:67)
        at
        kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
        at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662)
        [2015-05-14 17:36:50,358] INFO Completed load of log my-replicated-topic-1 with log end offset 13 (kafka.log.Log)
        [2015-05-14 17:36:50,363] INFO Recovering unflushed segment 0 in log my-replicated-topic-2.(kafka.log.Log)
        [2015-05-14 17:36:50,367] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
        java.lang.NullPointerException
        at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
        at kafka.log.Log.loadSegments(Log.scala:179)
        at kafka.log.Log.<init>(Log.scala:67)
        at
        kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
        at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
        [2015-05-14 17:36:50,371] INFO Completed load of log my-replicated-topic-2 with log end offset 14 (kafka.log.Log)
      2. 查看topic 描述信息时:
        D:kafka_2.11-0.8.2.1in>kafka-topics.bat --describe --zookeeper localhost:2181 --topic my-replicated-topic
        [2015-05-14 17:40:25,209] ERROR Unable to open socket to
        0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)
        [2015-05-14 17:40:25,210] WARN Session 0x0 for server null, unexpected error, closing socket
        connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
        java.net.SocketException: Address family not supported by protocol family: connect
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:532)
        at
        org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
        at
        org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
        Topic:my-replicated-topic PartitionCount:3 ReplicationFactor:2 Configs:
        Topic: my-replicated-topic Partition: 0 Leader: 9093 Replicas:9094,9093 Isr: 9093,9094
        Topic: my-replicated-topic Partition: 1 Leader: 9093 Replicas: 9093,9094 Isr: 9093,9094
        Topic: my-replicated-topic Partition: 2 Leader: 9093 Replicas: 9094,9093 Isr: 9093,9094
      3. 生产topic 消息时
        D:kafka_2.11-0.8.2.1in>kafka-console-producer.bat --broker-list localhost:9093 --topic my-replicated-topic
        [2015-05-14 17:42:27,943] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
      4. 消费topic 消息时
        D:kafka_2.11-0.8.2.1in>kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
        [2015-05-14 17:47:11,588] ERROR Unable to open socket to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)
        [2015-05-14 17:47:11,589] WARN Session 0x0 for server null, unexpected error, closing socket
        connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
        java.net.SocketException: Address family not supported by protocol family: connect
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:532)
        at
        org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
        at
        org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

    安装过程遇到的问题:

      1. kafka启动后提示
        Unrecognized VM option ‘+UseCompressedOops‘ Could not create the Java virtual machine.
        开始以为是内存大小的问题, 后来发现不是,是JDK的问题, 我用的32位centos,jdk1.6_24,  换成JDK1.7依然报错。
        查看 bin/kafka-run-class.sh  找到
        if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
          KAFKA_JVM_PERFORMANCE_OPTS="-server  -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
          -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" fi
        去掉-XX:+UseCompressedOops
        JMX_PORT=9999 bin/kafka-server-start.sh config/server.properties &
        启动成功
  • 相关阅读:
    js浏览器对象
    js对象
    js基础
    css的常见操作
    css 盒子模型
    css定位与浮动
    汇编三(习题)
    pyEcharts安装及使用指南(最新)
    pyEcharts安装及使用指南
    Anaconda安装第三方库与pip和conda 添加国内源
  • 原文地址:https://www.cnblogs.com/wzxblog/p/5731632.html
Copyright © 2020-2023  润新知