• 需登录账号与密码的网页爬取demo


    public static String connect(String dataUrl){     
    		String result = null;
    		try {
    			HttpClient httpclient = new DefaultHttpClient();
    			HttpPost httpPost = new HttpPost("http://rs.260g.com:8090/index.php?c=user&a=login");
    			httpPost.setHeader("User-Agent",
    					"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
    			// 创建名/值组列表 
    			List<NameValuePair> parameters = new ArrayList<NameValuePair>();
    			parameters.add(new BasicNameValuePair("username", "******"));
    			parameters.add(new BasicNameValuePair("password", "*****"));
    			parameters.add(new BasicNameValuePair("rememberMe", "****"));
    			// 创建UrlEncodedFormEntity对象 
    			UrlEncodedFormEntity formEntiry = new UrlEncodedFormEntity(parameters);
    			httpPost.setEntity(formEntiry);
    			HttpResponse httpResponse = httpclient.execute(httpPost);
    			HttpEntity entity = httpResponse.getEntity();			
    			result = IOUtils.toString(entity.getContent());
    			HttpGet searchHttpGet = new HttpGet(dataUrl);
    			searchHttpGet.setHeader("User-Agent",
    					"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
    			httpResponse = httpclient.execute(searchHttpGet);
    			result = IOUtils.toString(httpResponse.getEntity().getContent());
    		} catch (Exception e) {
    			e.printStackTrace();
    		}
    		return result;    
    } 
    

      

  • 相关阅读:
    leetcode 62. Unique Paths
    leetcode 345. Reverse Vowels of a String
    leetcode 344. Reverse String
    Centos7.4 kafka集群安装与kafka-eagle1.3.9的安装
    yarn调度器 FairScheduler 与 CapacityScheduler
    Hive性能优化
    HBase笔记
    Zookeeper笔记
    Hadoop组件详解(随缘摸虾)
    ubuntu18.04.2 hadoop3.1.2+zookeeper3.5.5高可用完全分布式集群搭建
  • 原文地址:https://www.cnblogs.com/ipetergo/p/7380535.html
Copyright © 2020-2023  润新知