采用Python3环境,并通过pip装几个包:
大概是我的系统环境变量没有设置好吧,我看网上都是直接pip install xxx,我的还要多加python -m 指令
设置好后,在python文件夹的Scripts里找到jupyter notebook.exe,打开,复制地址换浏览器需要输入服务器密码,按照网上教程我也没有修改成功,不知道原因,待解决吧。
右上角New-python3,开始编写属于你的第一个爬虫吧。
简单爬新浪网:
import requests res = requests.get('http://mobile.sina.com.cn/') res.encoding = 'utf-8' print(res.text)
这里get的原理在 http://study.163.com/my 网络爬虫实战中讲到,选择Chrome浏览器,右键检查,选择Doc,在Preview可以查看内容(需要F5重新爬取),是否爬取到内容
其他小试:
from bs4 import BeautifulSoup html_sample = ' <html> <head> <meta charset="utf-8"> <title></title> </head> <body> <h1> This is a test</h1> </body> </html>' soup = BeautifulSoup(html_sample,'html.parser') header = soup.select('h1') print(header) print(header[0].text)
from bs4 import BeautifulSoup html_sample = ' <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title></title> </head> <body> <h1 id = "title">C++</h1> <h1 id = "title">Python</h1> <a href="#" class = "link"> This is link2 </a> </body> </html> ' soup = BeautifulSoup(html_sample,'html.parser') alink = soup.select('#title') print(alink) print(' ') for link in alink: print(link) print(' ') for link in alink: print(link.text)
输出:
[<h1 id="title">C++</h1>, <h1 id="title">Python</h1>] <h1 id="title">C++</h1> <h1 id="title">Python</h1> C++ Python