• HTTP请求到爬虫代码的终南捷径


    前阵子在做爬虫的时候学会了各种抓包,看到http请求的时候硬拼代码实在有点累.
    后来发现Postman工具是直接可以把Postman请求直接生成对应的代码,这样一下来就美滋滋了.

    那么最后的问题就成了,抓包的网络请求怎么转成Postman 请求了.

    然后....

    在Chrome开发者工具里面又找到了好东西.

    首先打开需要的网站,我们以https://haojia.m.smzdm.com/tianmaoquan/ 为例.

    F12到开发者模式-Network,接着刷新一下页面观察网络请求.

    我们可以发现其中请求如下图:

    很明显这就是我们要的请求了.

    接着...

    选中对应的请求,右键 Copy - Copy as cURL

    然后大概拿到了下文一串cURL命令:

    curl 'https://haojia.m.smzdm.com/tianmaoquan/get_data?category=zonghe&order=CouponConsumeCount&sort=&page=1&keyword=' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,da;q=0.7' -H 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1' -H 'Accept: application/json, text/plain, */*' -H 'Referer: https://haojia.m.smzdm.com/tianmaoquan/' -H 'Cookie: __ckguid=k3u6YR5FixgKfa62bIyUga6; _ga=GA1.2.679944293.1516270199; device_id=17532887221516270201203456ed594c266fd84551f01a413af71d5a6f; smzdm_user_source=875817C616D1ED42EB82F39DFE768D0B; __gads=ID=3e3801168e373540:T=1518695168:S=ALNI_MZfywByvg_Ynjo4BpUmdXH30U9wOg; wt3_eid=%3B999768690672041%7C2152247098000283145%232152247111500241744; __jsluid=78d672b28c72540c578b65d9e9ca98cb; _ga=GA1.3.679944293.1516270199; zdm_qd=%7B%7D; _gid=GA1.2.1251205158.1523178568; from=wap; Hm_lvt_9b7ac3d38f30fe89ff0b8a0546904e58=1522470824,1523178567,1523178905; Hm_lpvt_9b7ac3d38f30fe89ff0b8a0546904e58=1523178914' -H 'Connection: keep-alive' --compressed
    

    放命令行跑一下,能看到正常返回:

    接着我们回到Postman,选择Import

    切到 Paste Raw Text

    把cURL命令扔进去

    点击一下Import,完整的请求自动导入了.

    试一下:

    然后....

    
    var client = new RestClient("https://haojia.m.smzdm.com/tianmaoquan/get_data?category=zonghe&order=CouponConsumeCount&sort=&page=1&keyword=");
    var request = new RestRequest(Method.GET);
    request.AddHeader("postman-token", "bec86b0b-92d9-6903-8b18-25cad8b1f5cd");
    request.AddHeader("cache-control", "no-cache");
    request.AddHeader("connection", "keep-alive");
    request.AddHeader("cookie", "__ckguid=k3u6YR5FixgKfa62bIyUga6; _ga=GA1.2.679944293.1516270199; device_id=17532887221516270201203456ed594c266fd84551f01a413af71d5a6f; smzdm_user_source=875817C616D1ED42EB82F39DFE768D0B; __gads=ID=3e3801168e373540:T=1518695168:S=ALNI_MZfywByvg_Ynjo4BpUmdXH30U9wOg; wt3_eid=%3B999768690672041%7C2152247098000283145%232152247111500241744; __jsluid=78d672b28c72540c578b65d9e9ca98cb; _ga=GA1.3.679944293.1516270199; zdm_qd=%7B%7D; _gid=GA1.2.1251205158.1523178568; from=wap; Hm_lvt_9b7ac3d38f30fe89ff0b8a0546904e58=1522470824,1523178567,1523178905; Hm_lpvt_9b7ac3d38f30fe89ff0b8a0546904e58=1523178914");
    request.AddHeader("referer", "https://haojia.m.smzdm.com/tianmaoquan/");
    request.AddHeader("accept", "application/json, text/plain, */*");
    request.AddHeader("user-agent", "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1");
    request.AddHeader("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,da;q=0.7");
    request.AddHeader("accept-encoding", "gzip, deflate, br");
    IRestResponse response = client.Execute(re
    

    完事...

    同理使用Wireshark、Charles都有类似转cURL的功能,能拿到cURL命令都可以这样玩。

  • 相关阅读:
    阅读笔记
    学习小记
    networkx学习笔记
    ORM查询简化
    redis等缓存
    redis相关缓存知识
    Centos7网络配置
    redis安装详细
    redis安装详细
    mobaxterm使用手册
  • 原文地址:https://www.cnblogs.com/liguobao/p/8736095.html
Copyright © 2020-2023  润新知