帮朋友写一个爬取地区信息的脚本

需求:

项目内容：客户要求整理如下：
第一种
原始数据
xx市白云区新市新街新巷16号
直接输出
xx市，白云区，新市新街新巷16号
第二种：
原始数据：
大沙地沙边街
输出数据：
xx市，黄埔区，大沙地沙边街

附加要求．将原始数据输出到第一列

提供的数据如下：

xx市白云区新市新街新巷16号
xx市花都区狮岭镇岭南工业园合和东路10号
xx市增城区派潭镇大埔村牛角塘一巷
xx市白云区同德街道同嘉路诚德大厦
荔湾区南岸铁路边7号顺景楼
xx市天河区车陂街道车陂高地大街
大沙地沙边街
寺右一马路96号201房
xx市海珠区龙凤街道革新路80号
xx市增城区新塘镇沙埔镇港口村

调用ＡＰＩ进行爬取
　　http://api.map.baidu.com/place/v2/search?q=%s&region=xx市&output=json&ak=vCx0pfB4y3UNeno7INcCi5wCSv4Gqaij

　　 

 有些是街道，就要通过街道去获取其所在区号．市倒是不用担心因为都是广东．

先开始写一个函数尝试爬取

#!/usr/bin/env python
#encoding=utf-8
#by i3ekr

import requests,re,time,json

success_list = []
def shell(values):
    json_data = json.loads(requests.get("http://api.map.baidu.com/place/v2/search?q=%s&region==xx市&output=json&ak=vCx0pfB4y3UNeno7INcCi5wCSv4Gqaij" % (values)).content)
    print json_data
    try:
        for n in range(0, len(json_data) + 1):
            c2 = json_data['results'][n]['area']
            c1 = u'xx市'
            c3 = values.decode('utf-8')
            if c1 in c3:
                c3 = c3.replace(c1, "")
            if c2 in c3:
                c3 = c3.replace(c2, "")
                success_list.append(c1 + "," + c2 + "," + c3)
            print c2
            break
    except Exception as e:
        print "error"

刚开始的时候我爬取的数据json格式是固定的

c2 = json_data['results'][1]['area']

后来发现这个area并不全都在第一个数据里．所以选择了先获取results的长度然后再进行结合try遍历，如果获取到就正常得到area并且break跳出循环遍历．

最后就是将这个函数进行封装然后进行利用即可．

最终得到代码如下：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# by i3ekr
#api_1 = vCx0pfB4y3UNeno7INcCi5wCSv4Gqaij
#api_2 = i1tGx6jjU3qFkeylf3S7ejBAoiQ6o91B
import json
import requests
import time
import sys

fail_list = []
success_list = []
def guolv(values):
    json_data = json.loads(requests.get("http://api.map.baidu.com/place/v2/search?q=%s&region=广州市&output=json&ak=vCx0pfB4y3UNeno7INcCi5wCSv4Gqaij" % (values)).content)
    try:
        for n in range(0, len(json_data) + 1):
            c2 = json_data['results'][n]['address']
            c1 = u'广州市'
            c3 = values.decode('utf-8')
            if c1 in c3:
                c3 = c3.replace(c1, "")
            if c2 in c3:
                c3 = c3.replace(c2, "")
            success_list.append(c1 + "," + c2 + "," + c3)
            break
    except Exception as e:
        fail_list.append(values)

def address(values):
    try:
        guolv(values)
    except Exception as e:
        fail_list.append(values)


def shell(values):
    if "广州市" in values and "区" in values:
        data = values.replace('广州市', '广州市,')
        success_list.append(data.replace('区', '区,'))
    elif "街" in values:
        jiedao_left = values.split('街')[0] + "街"
        jiedao_all = values
        try:
            guolv(jiedao_left)
        except Exception as e:
            address(values)
    else:
        guolv(values)

if __name__ == "__main__":
    with open("data.txt", "r+") as f:
        lines = f.readlines()
        now_time = time.time()
        for i in lines:
            data = i.strip("
")
            print(u"[+] 正在测试：　%s" % (data))
            shell(data)



    print(u"success %s" % (len(success_list)))
    print(u"fail    %s" % (len(fail_list)))
    print(u"----------")
    print(u'总共用时：%s'%(time.time() - now_time))

    for i in success_list:
        with open('success.txt','a+') as f:
            f.write(i+"
")
    with open('success.txt','a+') as f:
        f.write("[-]以下是失败的----------------------------------------------------
")
    for x in fail_list:
        with open('success.txt','a+') as f:
            f.write("[-]" + x + "
")