Scrapy是用python编写的爬虫程序。
Scrapyd是一个部署与运行scrapy爬虫的应用，提供JSON API的调用方式来部署与控制爬虫 。
本文验证在fedora与centos是安装成功。
运行需要如下程序：
l python2.7
l pip setuptools
l lxml
l OpenSSL(pyopenssl)
其中scrapyd需占用6800端口
安装依赖：
yum -y install readline-devel
yum -y install openssl-devel
yum -y install sqlite sqlite-devel
安装python2.7
wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz
tar zxvf Python-2.7.9.tgz
cd Python-2.7.9
./configure --with-zlib-dir=/usr/local/lib
make && make install
安装 pip：
wget https://bootstrap.pypa.io/get-pip.py
python2.7 get-pip.py
顺便也会安装 setuptools
安装lxml及依赖
yum install libffi-devel
yum install libxml2
yum install libxslt
yum install libxml2-devel
yum install libxslt-devel
pip install lxml
安装pyOpenSSL
https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6 --no-check-certificate
tar zxvf pyOpenSSL-0.15.1.tar
cd pyOpenSSL-0.15.1
python2.7 setup.py install
安装Twisted
yum -y install bzip2-devel
tar xf Twisted-14.0.0.tar.bz2
cd Twisted-14.0.0
python2.7 setup.py install
安装scrapy:
pip install Scrapy
测试安装：
scrappy startproject testProject， 当前目录下若能生成scrapy格式的路径，说明安装成功
安装scrapyd
pip install scrapyd
安装 scrapyd-client
wget https://github.com/scrapy/scrapyd-client/archive/master.zip
unzip master.zip
cd scrapyd-client-master
python2.7 setup.py install
scrapyd-client主要是便于在服务端用脚本来部署scrapy程序：scrapyd-deploy.