基本上是按官方文档所写 https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/，过完。·BeautifulSoup 内容太多了，用的时候查起来方便一点
开始使用呢
- 从一个
soup对象开始，以下两种方式生成一个soup对象
from bs4 import BeautifulSoup
soup = BeautifulSoup(open("index.html")) ##传入文件
soup = BeautifulSoup("<html>data</html>") ##文本
- 构造soup对象时，可以传入解析器参数，如果不传入的话，会以最好的方式去解析
解析器
解析器有如下
html.parser
lxml
xml
html5lib
- 使用
html.parser解析器
soup = BeautifulSoup("<html>data</html>","html.parser")
四种对象
Beautiful Soup将复杂HTML文档转换成一个复杂的树形结构,每个节点都是Python对象,所有对象可以归纳为4种 Tag , NavigableString , BeautifulSoup , Comment
tag对象
tag对象，同网页中的标签的意思
- html标签
soup = BeautifulSoup("<html>data</html>","html.parser")
soup.html
- a标签
soup = BeautifulSoup("<a >data</a>","html.parser")
soup.a
标签名(name)
标签拥有自己的名字，用可以直接使用标签调用
soup = BeautifulSoup("<html><a >data</a></html>","html.parser")
print(soup.a.name)
**结果为 a **
属性值(Attributes)
soup = BeautifulSoup("<html><a href='baidu.com'>data</a></html>","html.parser")
print(soup.a['href'])
**结果为： baidu.com **
多值属性
HTML 4定义了一系列可以包含多个值的属性.在HTML5中移除了一些,却增加更多.最常见的多值的属性是 class (一个tag可以有多个CSS的class). 还有一些属性 rel , rev , accept-charset , headers , accesskey . 在Beautiful Soup中多值属性的返回类型是list:
soup = BeautifulSoup("<html><a href='baidu.com' rev='ll' class='night'>data</a></html>","html.parser")
print(soup.a['class'])
print(soup.a['rev'])
**结果为： **
['night']
['ll']
非多值的情况下回返回字符串
id_soup = BeautifulSoup('<p id="my id"></p>')
id_soup.p['id']
# 'my id'
属性赋值，属性可以赋值，多值的情况如下，数组回以空格拼接上去
rel_soup = BeautifulSoup('<p>Back to the <a rel="index">homepage</a></p>')
rel_soup.a['rel']
# ['index']
rel_soup.a['rel'] = ['index', 'contents']
print(rel_soup.p)
# <p>Back to the <a rel="index contents">homepage</a></p>
内容
- string
刚刚我们已经得到了tag，并且获取到了属性，现在我们来获取内容。很简单tag元素调用.string
from bs4 import BeautifulSoup
if __name__ == '__main__':
soup = BeautifulSoup("<a >data</a>", "html.parser")
print(soup.a.string)
print(type(soup.a.string))
print(str(soup.a.string))
**结果为： **
data
<class 'bs4.element.NavigableString'>
data
我们发现这样得到的内容并不是str型，但查看bs4.element.NavigableString的源码，发现NavigableString的源码继承了str类
class NavigableString(str, PageElement):
省略其他代码
官方手册中如下描述如果想在Beautiful Soup之外使用 NavigableString 对象,需要调用 unicode() 方法,将该对象转换成普通的Unicode字符串,否则就算Beautiful Soup已方法已经执行结束,该对象的输出也会带有对象的引用地址.这样会浪费内存.而python3中并没有unicode()，使用str()代替
- text
text属性更常用，并且它可以直接过滤掉注释，而如果标签中属性，string啥也不会返回
markup = "<b><!--Hey, buddy. Want to buy a used parser?-->xxxx</b>"
soup = BeautifulSoup(markup,"html.parser")
print(type(soup.b.text))
print(soup.b.text)
**结果为： **
<class 'str'>
xxxx
Comment对象
这种情况下，会产生Comment对象
markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))
结果为：
Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>
我们可以看到这时候.string返回的对象不再是bs4.element.NavigableString，而是Comment
prettify()方法
prettify()方法返回html对象
markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))
print(soup.b.prettify())
结果为：
Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>
<b>
<!--Hey, buddy. Want to buy a used parser?-->
</b>
soup = BeautifulSoup("<a href='baidu.com'>data</a>", "html.parser")
print(soup.a.prettify())
结果为：
<a href="baidu.com">
data
</a>
find_all方法
返回所有<a> 标签列表
contents属性
返回元素的所有子元素 也是标签列表
el = soup.head.contents
children属性
返回所有子元素的一个迭代器
from bs4 import BeautifulSoup
html='''<!DOCTYPE html>
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
</body>
</html>'''
if __name__ == '__main__':
soup = BeautifulSoup(html, "html.parser")
al=soup.find_all('a')
print(al)
print(type(al[0]))
el = soup.head.contents
print(el)
print(type(el[0]))
chd=soup.head.children
print(chd)
for el in soup.head.children:
print(type(el))
结果为：
[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
<class 'bs4.element.Tag'>
[<title>The Dormouse's story</title>]
<class 'bs4.element.Tag'>
<list_iterator object at 0x0000021EABA5F2B0>
<class 'bs4.element.Tag'>
descendants属性
contents属性和children属性都是直接子元素，而descendants是所有子孙元素（这里看做孙元素）
from bs4 import BeautifulSoup
if __name__ == '__main__':
soup = BeautifulSoup("<html><head><title>The Dormouse's story</title></head></html>","html.parser")
head_tag = soup.head
for child in head_tag.descendants:
print(child)
结果为：
<title>The Dormouse's story</title>
The Dormouse's story
兄弟元素
from bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
</body>
</html>'''
if __name__ == '__main__':
soup = BeautifulSoup(doc, "html5lib")
print(soup.p)
print(soup.p.next_sibling)
下一个兄弟元素，兄弟元素的理解为同父元素，next_sibling下一个挨着的兄弟元素
结果为：
<p class="title"><b>The Dormouse's story</b></p>
//空行
解释一下上面的结果，上面空行是因为第一个p标签后面有回车符，所以p的下一个兄弟元素并不是第二个p。修改代码为如下
next_sibling
from bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p><p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
</body>
</html>'''
if __name__ == '__main__':
soup = BeautifulSoup(doc, "html.parser")
print(soup.p)
print(soup.p.next_sibling)
结果为：
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
previous_siblings
这个代表上一个兄弟元素，和next_sibling同理
next_element
next_element代表下一个元素，和下一个兄弟元素的 next_sibling相比，next_element不排除下一个不是兄弟元素的情况
last_a_tag = soup.find("a", id="link3")
last_a_tag
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
last_a_tag.next_sibling
# '; and they lived at the bottom of a well.'
last_a_tag.next_element
# u'Tillie'
previous_element
previous_element 代表上一个元素，和next_element同理
过滤器find_all(flter)
find_all 方法中的过滤器可以用来过滤tag中的name
字符串
soup.find_all('b')
# 结果为 [<b>The Dormouse's story</b>]
正则表达式
import re
for tag in soup.find_all(re.compile("^b")):
print(tag.name)
匹配到了所有b为开头的标签
结果为：
body
b
for tag in soup.find_all(re.compile("t")):
print(tag.name)
# 结果为下面两行，标签名包含t的标签
# html
# title
列表
soup.find_all(["a", "b"])
# [<b>The Dormouse's story</b>,
# <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
True/False
没太大意义和使用价值
for tag in soup.find_all(True):
print(tag.name)
# html
# head
# title
# body
# p
# b
# p
# a
# a
# a
# p
方法
定义了元素含有class属性，但不含id属性的过滤方法
def has_class_but_no_id(tag):
return tag.has_attr('class') and not tag.has_attr('id')
soup.find_all(has_class_but_no_id)
# 结果如下三行
# [<p class="title"><b>The Dormouse's story</b></p>,
# <p class="story">Once upon a time there were...</p>,
# <p class="story">...</p>]
方法还能过滤属性
def not_lacie(href):
return href and not re.compile("lacie").search(href)
soup.find_all(href=not_lacie)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
总概
soup.find_all("title")
# [<title>The Dormouse's story</title>]
soup.find_all("p", "title")
# [<p class="title"><b>The Dormouse's story</b></p>]
soup.find_all("a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.find_all(id="link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
import re
soup.find(string=re.compile("sisters"))
# u'Once upon a time there were three little sisters; and their names were
'
有几个方法很相似,还有几个方法是新的,参数中的 string 和 id 是什么含义?
为什么 find_all("p", "title") 返回的是CSS Class为”title”的<p>标签?
参数列表解读
find_all( name , attrs , recursive , string , **kwargs )
- name 根据tag的name筛选
- attrs 接受字典类型的属性比如
soup.find_all(attrs={"id": "title"})，阅读SoupStrainer类的源码可以发现如下
if not isinstance(attrs, dict):
# Treat a non-dict value for attrs as a search for the 'class'
# attribute.
kwargs['class'] = attrs
attrs = None
这里可以看出当attrs不为字典，为普通字符实，会把attrs这个参数当做'class'属性
- recursive 是否从当前位置递归往下查询，如果不递归，只会查询当前
soup文档的子元素 - string 这里是通过tag的内容来搜索，并且返回的是类容，而不是tag类型的元素
**kwargs自动拆包接受属性值，所以才会有soup.find_all('a',id='title')，id='title'为**kwargs自动拆包掺入
class_
按照CSS类名搜索tag的功能非常实用,但标识CSS类名的关键字 class 在Python中是保留字,使用 class 做参数会导致语法错误.从Beautiful Soup的4.1.1版本开始,可以通过 class_ 参数搜索有指定CSS类名的tag:
soup.find_all("a", class_="sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
搜索 class 属性时也可以通过CSS值完全匹配:
css_soup.find_all("p", class_="body strikeout")
# [<p class="body strikeout"></p>]
limit 参数
限制返回的条数
soup.find_all("a", limit=2)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
find
同find_all方法类似，唯一区别是find返回一个元素，find_all返回一个列表。方法参数和find_all相同find( name , attrs , recursive , string , **kwargs ) ，使用方式也相同
soup.find_all('title', limit=1)
# [<title>The Dormouse's story</title>]
soup.find('title')
# <title>The Dormouse's story</title>
除了find_all，find之外，还有很多方法类似，如下
find_parents
find_parents( name , attrs , recursive , string , **kwargs )
查找所有的父辈节点（包括多级的祖先），返回一个列表
find_parent
find_parent( name , attrs , recursive , string , **kwargs )
查找上一个父辈节点，会一直在文档树上面查找，不会只查找一级别，特别注意
from bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<div class='cover'> <div> <p></p> </div> </div>
</body>
</html>'''
if __name__ == '__main__':
soup = BeautifulSoup(doc, "html.parser")
p=soup.find('p')
print(p.find_parent("div"))
print(p.find_parents("div"))
#结果
<div> <p></p> </div> #文档树上一个查找的div
[<div> <p></p> </div>, <div class="cover"> <div> <p></p> </div> </div>] #所有div
官方案例
a_string = soup.find(string="Lacie")
a_string
# u'Lacie'
a_string.find_parents("a")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
a_string.find_parent("p")
# <p class="story">Once upon a time there were three little sisters; and their names were
# <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
# and they lived at the bottom of a well.</p>
a_string.find_parents("p", class="title")
# []
find_next_siblings，find_next_sibling
find_next_siblings( name , attrs , recursive , string , **kwargs )find_next_sibling( name , attrs , recursive , string , **kwargs )
这2个方法通过 .next_siblings 属性对当tag的所有后面解析 [5] 的兄弟tag节点进行迭代，find_next_siblings() 方法返回所有符合条件的后面的兄弟节点, find_next_sibling() 只返回符合条件的后面的第一个tag节点.
find_previous_siblings，find_previous_sibling
- find_previous_siblings( name , attrs , recursive , string , **kwargs )
- find_previous_sibling( name , attrs , recursive , string , **kwargs )
和find_next_siblings，find_next_sibling 类似，只是查找前面的兄弟元素
find_all_next，find_next
find_all_next( name , attrs , recursive , string , **kwargs )find_next( name , attrs , recursive , string , **kwargs )
这2个方法通过 .next_elements 属性对当前tag的之后的 [5] tag和字符串进行迭代, find_all_next() 方法返回所有符合条件的节点, find_next() 方法返回第一个符合条件的节点
find_all_previous，find_previous
find_all_previous( name , attrs , recursive , string , **kwargs )find_previous( name , attrs , recursive , string , **kwargs )
和find_all_next，find_next类似,向前查找。这2个方法通过 .previous_elements 属性对当前节点前面 [5] 的tag和字符串进行迭代。
Css选择器
select
css选择器的方法为select(css_selector)
目前支持的选择器如下案例，css选择器可以参考https://www.w3school.com.cn/cssref/css_selectors.ASP
soup.select("title")
# [<title>The Dormouse's story</title>]
soup.select("p nth-of-type(3)")
# [<p class="story">...</p>]
soup.select("body a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select("html head title")
# [<title>The Dormouse's story</title>]
soup.select("head > title")
# [<title>The Dormouse's story</title>]
soup.select("p > a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select("p > a:nth-of-type(2)")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
soup.select("p > #link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]
soup.select("body > a")
soup.select("#link1 ~ .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select("#link1 + .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
soup.select(".sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select("[class~=sister]")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select("#link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]
soup.select("a#link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
soup.select("#link1,#link2")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
soup.select('a[href]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select('a[href="http://example.com/elsie"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]
soup.select('a[href^="http://example.com/"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
# <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select('a[href$="tillie"]')
# [<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
soup.select('a[href*=".com/el"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]
multilingual_markup = """
<p lang="en">Hello</p>
<p lang="en-us">Howdy, y'all</p>
<p lang="en-gb">Pip-pip, old fruit</p>
<p lang="fr">Bonjour mes amis</p>
"""
multilingual_soup = BeautifulSoup(multilingual_markup)
multilingual_soup.select('p[lang|=en]')
# [<p lang="en">Hello</p>,
# <p lang="en-us">Howdy, y'all</p>,
# <p lang="en-gb">Pip-pip, old fruit</p>]
select_one
返回查找到的元素的第一个