• 安装ipython,使用scrapy shell来验证xpath选择的结果 | How to install iPython and how does it work with Scrapy Shell


    1. scrapy shell

    是scrapy包的一个很好的交互性工具,目前我使用它主要用于验证xpath选择的结果。安装好了scrapy之后,就能够直接在cmd上操作scrapy shell了。

    具体的解释参考官网

    https://docs.scrapy.org/en/latest/topics/shell.html

    2. ipython

    在官网推荐使用ipython来跑scrapy shell,于是我尝试安装。因为之前我的python环境都是通过conda配置的(参看前一篇),通过conda安装ipython很方便

    conda install -c conda-forge ipython

    然后整个ipython的包就会下载下来,因为都是编译好的,没有了烦人的编译失败的过程了。

    3. 运行ipython和在ipython上运行scrapy shell

    在当前的cmd运行框中,因为已经配置了系统环境,是可以直接运行python包的,所以直接在cmd运行框中敲入ipython就会进入到ipython的运行框,跟系统标准的cmd差不多,但是功能更丰富,色彩更丰富,布局也能好。

    但是当我直接在这上面敲scrapy shell命令,但是一直说没有此命令,失败。在这里卡住了。

    后来通过仔细阅读scrapy shell的说明

    If you have IPython installed, the Scrapy shell will use it (instead of the standard Python console). 

    意思就是scrapy shell会自己去找ipython的运行框。

    于是直接在cmd的标准运行框中输入scrapy shell <url>,返回的结果直接调用到ipython的运行框了。

    例子

    scrapy shell “http://scrapy.org” --nolog

    注意这里的双引号,之前我遵从官网的例子,写了单引号,一直返回invalid hostname,改为双引号之后,才解决问题。

    English Version

    1. scrapy shell

    is a good interactive tool that Scrapy provided. Currently I use it to validate xpath result before I write scripts. After installed Scrapy, the Scrapy shell can be used from CMD. 

    For detail specification, please refer to below

    https://docs.scrapy.org/en/latest/topics/shell.html

    2. ipython

    Scrapy recommand iPython as user interface to run Scrapy Shell, so I try to install iPython, which is also a Python extention package. As my Python env is from Conda, it's very easy to install iPython (how to install package in Conda env). Just run below command. 

    conda install -c conda-forge ipython

    3. Run iPython, and run Scrapy Shell on iPython

    How to run iPython? just key in "iPython" in your CMD prompt that has included Python path in system env path. A python-like user interface will appear, but more colorful, more functional and the layout is clearer. 

    So when I was in iPython interface, I try to key in "scrapy shell", but failed to work. The prompt said the command is not exist.

    I was confused, and stuck here. However I found below note when I read scrapy shell specification again, 

    If you have IPython installed, the Scrapy shell will use it (instead of the standard Python console). 

    So I just key in "scrapy shell <url>" in original windows CMD prompt, the scrapy shell automatically run at iPython interface 

    example

    scrapy shell “http://scrapy.org” --nolog

    Note, I use double quotation mark " " instead of single quotation mark ' ' which is used in scrapy shell specification example.

    it returns "invalid hostname" when I use single quotation mark.  

  • 相关阅读:
    linux查看硬件信息及驱动设备相关整理
    qt 画图相关
    glaux.h vs2008
    技巧:Vimdiff 使用
    Linux下显示硬件信息(二)lshw
    Linux下硬件信息查看(一)dmidecode
    qt delegate
    RFKill 【解决opensuse11.4 无法开启无线网的问题】
    板子上运行提示 找不到qml插件
    CRISPDM
  • 原文地址:https://www.cnblogs.com/wtang/p/7170670.html
Copyright © 2020-2023  润新知