• 数据结构化与保存


    1. 将新闻的正文内容保存到文本文件。

    def file(content):
        f = open('news.txt','a',encoding='utf-8')
        f.write(content)
        f.close()
    

    2. 将新闻数据结构化为字典的列表:

    • 单条新闻的详情-->字典news
    • 一个列表页所有单条新闻汇总-->列表newsls.append(news)
    • 所有列表页的所有新闻汇总列表newstotal.extend(newsls)
      import requests
      from bs4 import BeautifulSoup
      import datetime
      import re
      import pandas
      
      
      def getClickCount(newUrl):
          re1 = re.search('\_(.*).html',newUrl)
          re2 = re.match('http://news.gzcc.cn/html/2018/xiaoyuanxinwen_(.*).html',newUrl)
          i = re1.group(1).split('/')[-1]
          cUrl = 'http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80'.format(i)
          res = requests.get(cUrl)
          res2 = int(res.text.split(".html")[-1].lstrip("('").rstrip("');"))
          return(res2)
      
      def file(content):
          f = open('news.txt','a',encoding='utf-8')
          f.write(content)
          f.close()
      
      def getInformation(a1):
          res1 = requests.get(a1)
          res1.encoding = 'utf-8'
          soup1 = BeautifulSoup(res1.text, 'html.parser')
          new = {}
          new['title'] = soup1.select(".show-title")[0].text
          #new['content'] = soup1.select("#content")[0].text
          #file(new['content'])
          about = soup1.select('.show-info')[0].text
          time = about.lstrip('发布时间:')[:19]
          new['time'] = datetime.datetime.strptime(time, '%Y-%m-%d %H:%M:%S')
          if about.find('来源:') > 0:
              new['origin'] = about[about.find('来源:'):].split()[0].lstrip("来源:")
          else:
              new['origin'] = "未知"
          if about.find('作者:') > 0:
              new['writer'] = about[about.find('作者:'):].split()[0].lstrip("作者:")
          else:
              new['writer'] = "佚名"
          if about.find('审核:') > 0:
              new['audit'] = about[about.find('审核:'):].split()[0].lstrip("审核:")
          else:
              new['audit'] = "佚名"
          if about.find('摄影:') > 0:
              new['photograph'] = about[about.find('摄影:'):].split()[0].lstrip("摄影:")
          else:
              new['photograph'] = "佚名"
          new['url'] = a1
          new['count'] = getClickCount(a1)
          return(new)
      
      
      def getnewslist(url):
          resurl = requests.get(url)
          resurl.encoding = 'utf-8'
          soup = BeautifulSoup(resurl.text, 'html.parser')
          a = soup.select('li')
          list = []
          for news in a:
              if len(news.select('.news-list-title')) > 0:
                  a1 = news.select('a')[0].attrs['href']
                  list.append(getInformation(a1))
          return (list)
      
      def getPage(url):
          res = requests.get(url)
          res.encoding = 'utf-8'
          soup = BeautifulSoup(res.text,'html.parser')
          n = int(soup.select('.a1')[0].text.rstrip('条'))//10+1
          return(n)
      
      
      url = "http://news.gzcc.cn/html/xiaoyuanxinwen/"
      
      newTotal = []
      newTotal.extend(getnewslist(url))
      n = getPage(url)
      print(n)
      for i in range(n,n+1):
          urls = "http://news.gzcc.cn/html/xiaoyuanxinwen/{}.html".format(i)
          newTotal.extend(getnewslist(urls))
      

    3. 安装pandas,用pandas.DataFrame(newstotal),创建一个DataFrame对象df.

    4. 通过df将提取的数据保存到csv或excel 文件。

    dt = pandas.DataFrame(newTotal)
        dt.to_excel('new.xlsx')
    

      

    5. 用pandas提供的函数和方法进行数据分析:

    • 提取包含点击次数、标题、来源的前6行数据
    • 提取‘学校综合办’发布的,‘点击次数’超过3000的新闻。
    • 提取'国际学院'和'学生工作处'发布的新闻。
      print(dt.head(6))
      print(dt[(dt['count'] > 700)&(dt['origin']=='学校综合办')])
      sou = ['国际学院', '学生工作处']
      print(dt[dt['origin'].isin(sou)])
      

        

  • 相关阅读:
    java处理特殊时间格式,2019-11-28T06:52:09.724+0000 转为常见格式2019-11-28 06:52:09,同时转为数据库日期格式Timestamp
    最近比较忙,处理项目上各种问题。此时有新任务来临,赶时间记录一个方法,加深对数组的理解
    springboot项目通过gradle运用capsule插件打可执行jar包
    HanLP 关键词提取。总结
    mmdet阅读笔记
    mmcv阅读笔记
    Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
    3D Hand Shape and Pose from Images in the Wild
    End to end recovery of human shape and pose
    工作小结五
  • 原文地址:https://www.cnblogs.com/hano/p/8855031.html
Copyright © 2020-2023  润新知