storm本地python开发环境搭建

搭建storm本地python开发环境

需要使用python进行storm的开发，这里记录本地开发环境搭建,过程中使用的资源gitee中下载

• centos7 普通用户
• python 3.5.1
• jdk 1.8.0
• storm 1.1.0
• lein+streamparse

本地用户faron, 根目录/home/faron

这次使用普通用户安装

python 3.5.1安装

这里我之前使用pyenv已经部署好 /home/faron目录下的pthon环境为3.5.1, 这里可自行搜索

jdk 1.8.0安装

• 安装包下载

• 在～/.bashrc文间尾部追加java环境变量

  # java
  export JAVA_HOME=//home/faron/lein/java_storm/jdk1.8.0_161
  export JRE_HOME=$JAVA_HOME/jre
  # export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
  export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
  

• 激活环境变量

  source ~/.bashrc
  

• 校验java环境

  java versoin
  

  终端显示

  java version "1.8.0_161"
  Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
  Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

strom 1.1.0安装

• 安装包下载

• 在～/.bashrc文间尾部追加storm环境变量

  # storm
  export STORM_HOME=/home/faron/lein/java_storm/apache-storm-1.1.0
  export PATH=$STORM_HOME/bin:$PATH
  

• 激活环境变量

  source ~/.bashrc
  

• 校验storm环境

  storm version
  

  终端显示

  Running: //home/faron/lein/java_storm/jdk1.8.0_161/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/faron/lein/java_storm/apache-storm-1.1.0 -Dstorm.log.dir=/home/faron/lein/java_storm/apache-storm-1.1.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-core-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/kryo-3.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/reflectasm-1.10.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/asm-5.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/minlog-1.3.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/objenesis-2.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/clojure-1.7.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/ring-cors-0.1.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/disruptor-3.3.2.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-api-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-core-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-slf4j-impl-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/slf4j-api-1.7.21.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-over-slf4j-1.6.6.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/servlet-api-2.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-rename-hack-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/conf org.apache.storm.utils. VersionInfo
  Storm 1.1.0
  URL https://git-wip-us.apache.org/repos/asf/storm.git -r e40d213de7067f7d3aa4d4992b81890d8ed6ff31
  Branch (no branch)
  Compiled by ptgoetz on 2017-03-21T17:04Z
  From source with checksum 4ddc442e8b804654454ab26fbec6348

lein+streamparse安装

• lein安装

  安装过程中可能会出现lein脚本无法下载,leiningen-2.9.3-standalone.jar无法下载的情况

  makdir ~/bin
  cd ~/bin
  cp ~/lein/lein ./
  chmod +x ~/bin/lein
  # 运行lein安装脚本
  lein
  

  终端显示(下载速度可能会有点慢)

  WARNING: You have $CLASSPATH set, probably by accident.
  It is strongly recommended to unset this before proceeding.
  Downloading Leiningen to /home/faron/.lein/self-installs/leiningen-2.9.3-standalone.jar now...
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  100 637 100 637 0 0 393 0 0:00:01 0:00:01 --:--:-- 393
  27 13.9M 27 3942k 0 0 34711 0 0:07:02 0:01:56 0:05:06 26468

  如果leiningen-2.9.3-standalone.jar下载失败

  cp ~/lein/java_storm/leiningen-2.9.3-standalone.zip ~/.lein/self-installs/
  

  lein环境验证

  lein version
  # >>>终端显示
  # WARNING: You have $CLASSPATH set, probably by accident.
  # It is strongly recommended to unset this before proceeding.
  # Leiningen 2.9.3 on Java 1.8.0_161 Java HotSpot(TM) 64-Bit Server VM
  

• streamparse安装

  • 安装

    pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip
    pip install -i https://pypi.tuna.tsinghua.edu.cn/simple streamparse
    

  • 启动wordcount

    mkdir ~/storm_workplace
    cd ~/storm_workplace
    sparse quickstart wordcount
    cd wordcount
    sparse run
    

  • 异常中心包无法下载

    Could not transfer artifact org.apache.storm:storm-core:jar:1.1.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
    Could not find artifact org.apache.storm:storm-core:jar:1.1.0 in clojars (https://repo.clojars.org/)
    Could not transfer artifact org.clojure:clojure:jar:1.7.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
    Could not find artifact org.clojure:clojure:jar:1.7.0 in clojars (https://repo.clojars.org/)
    This could be due to a typo in :dependencies, file system permissions, or network issues.
    If you are behind a proxy, try setting the 'http_proxy' environment variable.

    # 仅限storm1.1.0，因为我的repository里面是storm-core:1.1.0本地环境
    tar -zxvf ~/lein/java_storm/repository.tar.gz -C  ~/.m2/
    # 再次运行
    sparse run