• pycharm环境下配置scrap爬虫环境


    【写在开头】

    参考文章后面给出了备注信息,是在解决这个问题的时候,查找的比较有亮点的参考文章,如果本文章写的不太清楚的,可以去原文章进行查看。下面列举的四个文章有参考的成分也有验证的成分,解决办法重点参考的是文章4,前面3个只是对pycharm使用和安装包的便捷性上比较有亮点,这里还是进行引用。后面方便对参考文章进行引用,对文献进行编号。

    【参考文章】

    1、https://www.cnblogs.com/zengsf/p/9801966.html

    pycharm的设置选项中安装文件,只是作为了解,知道可以通过pycharm平台安装和卸载包,该文章也介绍了不能直接安装twisted,所以还是需要下载安装,我是直接从文献4中给的百度盘下载安装的,twisted安装成功之后,可以成功安装scrapy包,但是不能查看scrapy版本,提示scrapy不是内部命令,放弃了,直接看正文部分教程或者文章4

    2、http://www.manongjc.com/article/44529.html

    提到的用豆瓣源进行加速安装的命令,对了一下,直接pip install scrapy和使用豆瓣源安装,后者的确很快,看了下https://pypi.douban.com/simple 源社区里面很多包,安装比较慢的时候可以考虑使用,作为了解

    3、https://www.cnblogs.com/llssx/p/8378832.html

    给出安装scrapy时会遇到的一些错误提示,和安装scrapy时自下而上的顺序结构图,只是作为了解,文章没有给出解决方案

    4、https://blog.csdn.net/YiXiao1997/article/details/96345737

    是一个完整的安装流程,并且出Twisted的百度盘文件,很方便。文章最后对该教程与该文章进行了对比。

    【正文部分】

    1、由于入手python的时候就用的pycharm,所以学到scrapy的时候就想着在pycharm下进行一些配置,pycharm有个控制台,见图1,可以直接输入命令行进行安装操作,本来以为直接pip install scrapy就可以了,但事实并不是这样,下图2给出了提示的错误信息,实际提示的错误信息有将近一千多行,直接去最后比较重要的两行,大概意思就是:安装twisted出现错误,Microsoft Visual C++ 14.0文件缺失。(备注:错误提示内容在文献3提到了)

    图1 pycharm控制台

    图2 错误提示信息:安装twisted出现错误,Microsoft Visual C++ 14.0文件缺失

    2、接着根据文献4的提示,下载好twisted的安装包,在pycharm的控制台进行安装程序,安装提示信息如下所示:

    D:zhuomian>pip install Twisted-19.2.0-cp37-cp37m-win_amd64.whl
    Processing d:zhuomian	wisted-19.2.0-cp37-cp37m-win_amd64.whl
    Requirement already satisfied: constantly>=15.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twiste
    d==19.2.0) (15.1.0)
    Requirement already satisfied: attrs>=17.4.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twisted==
    19.2.0) (19.1.0)
    Requirement already satisfied: PyHamcrest>=1.9.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twist
    ed==19.2.0) (1.9.0)
    Requirement already satisfied: zope.interface>=4.4.2 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from T
    wisted==19.2.0) (4.6.0)
    Requirement already satisfied: incremental>=16.10.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Tw
    isted==19.2.0) (17.5.0)
    Requirement already satisfied: Automat>=0.3.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twisted=
    =19.2.0) (0.7.0)
    Requirement already satisfied: hyperlink>=17.1.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twist
    ed==19.2.0) (19.0.0)
    Requirement already satisfied: six in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from PyHamcrest>=1.9.0->
    Twisted==19.2.0) (1.12.0)
    Requirement already satisfied: setuptools in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from PyHamcrest>=
    1.9.0->Twisted==19.2.0) (39.0.1)
    Requirement already satisfied: idna>=2.5 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from hyperlink>=17
    .1.1->Twisted==19.2.0) (2.8)
    Installing collected packages: Twisted
    Successfully installed Twisted-19.2.0

     3、安装好twisted之后,使用命令pip install scrapy继续安装scrapy,下面是控制台提示信息,成功安装。

    D:zhuomian>pip install scrapy
    Collecting scrapy
      Using cached https://files.pythonhosted.org/packages/a3/b1/d1ab5b3f84640097cf5ff642e2e357546781746d4fec2ebb40432904c57d/Scrapy-1.7.2-py2.py3-n
    one-any.whl
    Requirement already satisfied: lxml in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (4.3.4)
    Requirement already satisfied: pyOpenSSL in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (19.0.0)
    Requirement already satisfied: Twisted>=13.1.0; python_version != "3.4" in c:users余生,别无他求appdatalocalprogramspythonpython37libsi
    te-packages (from scrapy) (19.2.0)
    Requirement already satisfied: queuelib in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (1.5.0)
    Requirement already satisfied: six>=1.5.2 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from scrapy) (1.1
    2.0)
    Requirement already satisfied: service-identity in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from scrapy
    ) (18.1.0)
    Requirement already satisfied: cssselect>=0.9 in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (1.0.3)
    Requirement already satisfied: w3lib>=1.17.0 in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (1.20.0)
    Requirement already satisfied: PyDispatcher>=2.0.5 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from scr
    apy) (2.0.5)
    Requirement already satisfied: parsel>=1.5 in c:users余生,别无他求appdata
    oamingpythonpython37site-packages (from scrapy) (1.5.1)
    Requirement already satisfied: cryptography>=2.3 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from pyOpe
    nSSL->scrapy) (2.7)
    Requirement already satisfied: PyHamcrest>=1.9.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twist
    ed>=13.1.0; python_version != "3.4"->scrapy) (1.9.0)
    Requirement already satisfied: incremental>=16.10.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Tw
    isted>=13.1.0; python_version != "3.4"->scrapy) (17.5.0)
    Requirement already satisfied: hyperlink>=17.1.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twist
    ed>=13.1.0; python_version != "3.4"->scrapy) (19.0.0)
    Requirement already satisfied: Automat>=0.3.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twisted>
    =13.1.0; python_version != "3.4"->scrapy) (0.7.0)
    Requirement already satisfied: constantly>=15.1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twiste
    d>=13.1.0; python_version != "3.4"->scrapy) (15.1.0)
    Requirement already satisfied: zope.interface>=4.4.2 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from T
    wisted>=13.1.0; python_version != "3.4"->scrapy) (4.6.0)
    Requirement already satisfied: attrs>=17.4.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from Twisted>=
    13.1.0; python_version != "3.4"->scrapy) (19.1.0)
    Requirement already satisfied: pyasn1 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from service-identity
    ->scrapy) (0.4.5)
    Requirement already satisfied: pyasn1-modules in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from service-
    identity->scrapy) (0.2.5)
    Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from cryp
    tography>=2.3->pyOpenSSL->scrapy) (1.12.3)
    Requirement already satisfied: asn1crypto>=0.21.0 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from cryp
    tography>=2.3->pyOpenSSL->scrapy) (0.24.0)
    Requirement already satisfied: setuptools in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from PyHamcrest>=
    1.9.0->Twisted>=13.1.0; python_version != "3.4"->scrapy) (39.0.1)
    Requirement already satisfied: idna>=2.5 in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from hyperlink>=17
    .1.1->Twisted>=13.1.0; python_version != "3.4"->scrapy) (2.8)
    Requirement already satisfied: pycparser in c:users余生,别无他求appdatalocalprogramspythonpython37libsite-packages (from cffi!=1.11.3,
    >=1.8->cryptography>=2.3->pyOpenSSL->scrapy) (2.19)
    Installing collected packages: scrapy
    Successfully installed scrapy-1.7.2

    4、进行验证,查看scrapy版本,没有提示scrapy不是内部命令,也给出了版本信息,说明安装成功

    D:zhuomian>scrapy
    Scrapy 1.7.2 - no active project
    
    Usage:
      scrapy <command> [options] [args]
    
    Available commands:
      bench         Run quick benchmark test
      fetch         Fetch a URL using the Scrapy downloader
      genspider     Generate new spider using pre-defined templates
      runspider     Run a self-contained spider (without creating a project)
      settings      Get settings values
      shell         Interactive scraping console
      startproject  Create new project
      version       Print Scrapy version
      view          Open URL in browser, as seen by Scrapy
    
      [ more ]      More commands available when run from project directory
    
    Use "scrapy <command> -h" to see more info about a command

     【写在最后】

      就像前面说的,这篇文章是自己对在学习scrapy环境配置的时候,遇到的问题的一个记录,参考的还是网上的解决办法,但同时也是对网上的教程的一个验证,前面四篇文章的备注都作出了说明,对比一下第四篇参考文章,我这里没有安装wheel工具包,直接安装的twisted,也是可以成功安装的。

       写到这里,只是进行了环境配置,还没有进行验证,是不是可以成功进行项目开发,下一篇进行讲解,scrapy简单的项目开发。

     小编尊重原创,文章如有侵权,联系小编删帖,谢谢监督。转发请备注出处!

  • 相关阅读:
    luogu CF804D Expected diameter of a tree |Tarjan+樹的直徑+前綴和+二分查找
    luogu CF25C Roads in Berland |最短路floyd
    自觉日志·1·再现曾经的自觉
    祝福您新年快乐!
    自觉体验十一 体验祝福,与弘誓大愿!
    自觉体验 十 体验失忆
    自觉体验 六 手语与结印
    自觉体验 一  闲来无事,闭目养神
    我将离开屏幕前的“      ”
    第五章  自觉与【转载】《大念处经》
  • 原文地址:https://www.cnblogs.com/ljxh/p/11235079.html
Copyright © 2020-2023  润新知