文本处理

#文本长度
1 text1 = "Ethics are built right into the ideals and objectives of the United Nations "

len(text1) # The length of text1

76

#文本单词个数
text2 = text1.split(' ') # Return a list of the words in text2, separating by ' '.

len(text2)

14

text2

['Ethics',
 'are',
 'built',
 'right',
 'into',
 'the',
 'ideals',
 'and',
 'objectives',
 'of',
 'the',
 'United',
 'Nations',
 '']

通过列表查找单词

[w for w in text2 if len(w) > 3] # 查找单词长度大于3的单词

['Ethics',
 'built',
 'right',
 'into',
 'ideals',
 'objectives',
 'United',
 'Nations']

查找单词是开头的单词（即单词首字母大写）

[w for w in text2 if w.istitle()] # Capitalized words in text2

['Ethics', 'United', 'Nations']

查找以s结尾的单词

[w for w in text2 if w.endswith('s')] # Words in text2 that end in 's'

['Ethics', 'ideals', 'objectives', 'Nations']

使用set去重

text3 = 'To be or not to be'
text4 = text3.split(' ')

len(text4)

6

len(set(text4))

5

set(text4)

{'To', 'be', 'not', 'or', 'to'}

len(set([w.lower() for w in text4])) # 把单词转化为小写

4

set([w.lower() for w in text4])

{'be', 'not', 'or', 'to'}


处理自由文本

text5 = '"Ethics are built right into the ideals and objectives of the United Nations" 
#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'
text6 = text5.split(' ')

text6

['"Ethics',
 'are',
 'built',
 'right',
 'into',
 'the',
 'ideals',
 'and',
 'objectives',
 'of',
 'the',
 'United',
 'Nations"',
 '#UNSG',
 '@',
 'NY',
 'Society',
 'for',
 'Ethical',
 'Culture',
 'bit.ly/2guVelr']

[w for w in text6 if w.startswith('#')]

['#UNSG']

 

[w for w in text6 if w.startswith('@')]

['@']

text7 = '@UN @UN_Women "Ethics are built right into the ideals and objectives of the United Nations" 
#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'
text8 = text7.split(' ')

使用复杂的正则表达式

We can use regular expressions to help us with more complex parsing.

For example '@[A-Za-z0-9_]+' will return all words that:

• start with '@' and are followed by at least one:
• capital letter ('A-Z')
• lowercase letter ('a-z')
• number ('0-9')
• or underscore ('_')

import re # import re - a module that provides support for regular expressions

[w for w in text8 if re.search('@[A-Za-z0-9_]+', w)]

['@UN', '@UN_Women']