参考了DotNetSpider示例，
感觉DotNetSpider太重了，它是一个比较完整的爬虫框架。
对比了以下各种无头浏览器，最终采用PuppeteerSharp+AngleSharp写一个爬虫示例。
和上面的博文一样，都是用汽车之家的https://store.mall.autohome.com.cn/83106681.html这个页面做数据采集示例。
本文中使用PuppeteerSharp获取最终页面(即加载JavaScript之后的页面)，使用AngleSharp进行Html documents解析处理。
Headless Browsers
A list of (almost) all headless web browsers in existence
A web browser without a graphical user interface, controlled programmatically. Used for automation, testing, and other purposes.
Browser engines
These browser engines fully render web pages or run JavaScript in a virtual DOM
| Name | About | Supported Languages | License |
|---|---|---|---|
| Chromium Embedded Framework | CEF is a open source project based on the Google Chromium project. | JavaScript | BSD |
| Erik | Headless browser on top of Kanna and WebKit. | Swift | MIT |
| jBrowserDriver | A Selenium-compatible headless browser which is written in pure Java. WebKit-based. Works with any of the Selenium Server bindings. | Java | Apache License v2.0 |
| PhantomJS | [Unmaintained] PhantomJS is a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG. | JavaScript, Python, Ruby, Java, C#, Haskell, Objective-C, Perl, PHP, R(via Selenium) | BSD 3-Clause |
| Splash | Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python using Twisted and QT. | Any | BSD 3-Clause |
Multi drivers
These libraries can control multiple browser engines (typically using Selenium)
| Name | About | Supported Languages | License |
|---|---|---|---|
| CasperJS | CasperJS is an open source navigation scripting & testing utility written in Javascript for the PhantomJS WebKit headless browser and SlimerJS (Gecko). | JavaScript | MIT |
| Geb | Geb is a Groovy interface to WebDriver. | Groovy | Apache |
| Selenium | Selenium is a suite of tools to automate web browsers across many platforms. | JavaScript, Python, Ruby, Java, C#, Haskell, Objective-C, Perl, PHP, R | Apache |
| Splinter | Splinter is an open source tool for testing web applications using Python. It lets you automate browser actions, such as visiting URLs and interacting with their items. | Python | - |
| SST | SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests. | Python | - |
| Watir | The most elegant way to use Selenium WebDriver with ruby. | Ruby | MIT |
PhantomJS drivers
These libraries control PhantomJS
| Name | About | Supported Languages | License |
|---|---|---|---|
| Ghostbuster | Automated browser testing via phantom.js, with all of the pain taken out! That means you get a real browser, with a real DOM, and can do real testing! | JavaScript | Not specified |
| jedi-crawler | Lightsabing Node/PhantomJS crawler; scrape dynamic content : without the hassle | JavaScript | Not specified |
| Lotte | Lotte is a headless, automated testing framework built on top of PhantomJS and inspired by Ghostbuster. | JavaScript | MIT |
| phantompy | Phantompy is a headless WebKit engine with powerful pythonic api build on top of Qt5 Webkit | Python | LGPL-2.1 |
| X-RAY | Supports strings, arrays, arrays of objects, nested object structures, selector API, pagination, crawler, concurrency, throttles, delays, timeouts, and pluggable drivers (PhantomJS, HTTP) | JavaScript | MIT |
| Horseman | Promise based Node.js module for PhantomJS. Features chainable API, understandable control-flow, support for multiple tabs, and built-in jQuery. | JavaScript | MIT |
Chromium drivers
These libraries control Chromium
| Name | About | Supported Languages | License |
|---|---|---|---|
| Awesomium | Chromium-based headless browser engine | C++, | Free/Commercial |
| Headless Chromium | Chromium feature activated with the --headlesss flag, currently availible in the nightly build of Chromium, not yet released |
C++ | Opensource |
| Puppeteer | Headless Chrome Node API from the Chrome DevTools team | JavaScript | Apache |
| PuppeteerSharp | PuppeteerSharp is a port of the official Headless Chrome Node.JS Puppeteer API | MIT | |
| chrome-remote-interface | Chrome Debugging Protocol interface for Node.js | JavaScript | MIT |
| Chromy | Features chainable API, mobile emulation, fundamental API such as javascript evaluation. | JavaScript | MIT |
| chromedp | A faster, simpler way to drive browsers (Chrome, Edge, Safari, Android, etc) without external dependencies (ie, Selenium, PhantomJS, etc) using the Chrome Debugging Protocol. | Go | MIT |
| Chromeless | Chrome automation made simple. Runs locally or headless on AWS Lambda. | JavaScript | MIT |
Webkit drivers
These drivers control an in-process instance of Webkit
| Name | About | Supported Languages | License |
|---|---|---|---|
| Browserjet | Runs a custom build of webkit, controlled by node.js interface. | JavaScript | Not specified |
| ghost.py | ghost.py is a webkit web client written in python. | Python | MIT |
| headless_browser | Headless browser based on WebKit written in C++. | C++ | Not Specified |
| Jabba-Webkit | Jabba's headless webkit browser for scraping AJAX-powered webpages. | Python | Not specified |
| Jasmine-Headless-Webkit | jasmine-headless-webkit uses the QtWebKit widget to run your specs without needing to render a pixel. | Python, JavaScript, Ruby | Free |
| Python-Webkit | Python-Webkit is a python extension to Webkit to add full, complete access to Webkit's DOM | Python | GNU |
| Spynner | Programmatic web browsing module with AJAX support for Python | Python | Not specified |
| Webloop | Scriptable, headless WebKit with a Go API. | Go | BSD 3-Clause |
| wkhtmltopdf wkhtmltox wkhtmltoimage | Command line tool rendering HTML into PDF and other image formats. | shell, C | LGPLv3 |
| WKZombie | Functional headless browser (with JSON support) for iOS using WebKit and hpple/libxml2. | Swift | MIT |
Other drivers
These libraries control lesser known browsers or OS-provided web libraries
| Name | About | Supported Languages | License |
|---|---|---|---|
| Nightmare | Nightmare is a high-level browser automation library built as an easier alternative to PhantomJS. It runs on the Electron engine. | JavaScript | MIT |
| grope | A RubyCocoa interface to the macOS WebKit Framework | RubyCocoa | MIT |
| SlimerJS | SlimerJS is similar to PhantomJs, except that it runs Gecko, the browser engine of Mozilla Firefox, instead of Webkit (And it is not yet truly headless). | JavaScript | Mozilla 2.0 |
| SpecterJS | A scriptable headless Internet Explorer port of PhantomJS. | JavaScript | MIT |
| trifleJS | A headless Internet Explorer browser using the WebBrowser Class with a Javascript API running on the V8 engine. | JavaScript | MIT |
Fake Browser Engine
These libraries are typically naive or HTML-only browsers
| Name | About | Supported Languages | License |
|---|---|---|---|
| AngleSharp | Http Parsing Library | MIT | |
| Guillotine | A headless browser, written in C# | LGPL-3.0 | |
| benv | Stub a browser environment in node.js and headlessly test your client-side code. | JavaScript | MIT |
| browser.rb | Headless Ruby browser on top of Nokogiri and TheRubyRacer | Ruby | Not specified |
| BrowserKit | BrowserKit simulates the behavior of a web browser. | PHP | MIT |
| DamonJS | Bot navigating urls and doing tasks. | JavaScript | Apache |
| Headless | Headless browser support for fast web acceptance testing in | MIT | |
| HeadlessBrowser | A very miniature headless browser, for testing the DOM on Node.js | JavaScript | Not specified |
| HtmlUnit | HtmlUnit is a "GUI-Less browser for Java programs". | Java | Apache |
| Jaunt | Java Web Scraping & Automation API | Java | Not specified |
| JSDom | A JavaScript implementation of the WHATWG DOM and HTML standards, for use with Node.js. | JavaScript | MIT |
| MechanicalSoup | A Python library for automating interaction with websites. | Python | MIT |
| mechanize | Stateful programmatic web browsing. | Python | BSD 3-Clause, ZPL 2.1 |
| node-as-browser | Create a browser-like environment within Node.js | JavaScript | MIT |
| RoboBrowser | A simple, Pythonic library for browsing the web without a standalone web browser. | Python | BSD 3-Clause |
| SimpleBrowser | A flexible and intuitive web browser engine designed for automation tasks. Built on the 4 framework. | BSD 3-Clause | |
| stanislaw | Naive, mechanize-like HTML parser/form driver. | Python | Not specified |
| twill | Twill is a simple language that interacts with basic HTML pages (no JavaScript support). | Python | MIT |
| WeasyPrint | WeasyPrint is a visual rendering engine for HTML and CSS that can export to PDF. It aims to support web standards for printing. | Python | BSD 3-Clause |
| WWW::Mechanize | Headless browser for Perl with many plugins and extensions, notably Test::WWW:Mechanize for testing | Perl | Perl 5 |
| X-RAY | Supports strings, arrays, arrays of objects, nested object structures, selector API, pagination, crawler, concurrency, throttles, delays, timeouts, and pluggable drivers (PhantomJS, HTTP) | JavaScript | MIT |
| Xidel (Internet Tools) | An XQuery-based cli web scraper for static X/HTML pages and JSON-APIs. | FreePascal, XQuery | GPL-2 |
| Zombie.js | Zombie.js is a lightweight framework for testing client-side JavaScript code in a simulated environment. No browser required. | JavaScript | MIT |
Runs in a browser
| Name | About | Supported Languages | License |
|---|---|---|---|
| DalekJS | [unmaintained and recommend TestCafé] Automated cross browser testing with JavaScript. | JavaScript | MIT |
| TestCafé | Automated browser testing for the modern web development stack. | JavaScript | MIT |
| Sahi | Sahi is a cross-browser automation/testing tool with the facility to record and playback scripts. | JavaScript, Java, Ruby, PHP | Apache / Commercial |
| WatiN | Web Application Testing In | Apache 2.0 |
Misc tools
| Name | About | Supported Languages | License |
|---|---|---|---|
| browser-launcher | Detect and launch browser versions, headlessly or otherwise | JavaScript | MIT |
其实如果没有JavaScripts加载数据需求，单独用AngleSharp就可以搞定了。
但涉及到JavaScripts加载数据需求的，就需要上真正的无头浏览器组件才能搞定了。AngleSharp现在只支持简单的JavaScripts代码执行，稍微复杂点的，都不行，听说以后要完整支持JavaScripts，敬请期待吧！
Code
/*
* This is a Puppeteer+AngleSharp crawler console app samples
*/
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using AngleSharp;
using AngleSharp.Dom;
using AngleSharp.Html.Parser;
using Newtonsoft.Json;
using PuppeteerSharp;
namespace CrawlerSamples
{
internal class Program
{
private const string Url = "https://store.mall.autohome.com.cn/83106681.html";
private const int ChromiumRevision = BrowserFetcher.DefaultRevision;
private static async Task Main(string[] args)
{
//Download chromium browser revision package
await new BrowserFetcher().DownloadAsync(ChromiumRevision);
//Test AngleSharp
await TestAngleSharp();
Console.ReadKey();
}
private static async Task TestAngleSharp()
{
/*
* Used AngleSharp loading of HTML document
* TODO: Used WithJavaScript function need install AngleSharp.Scripting.Javascript nuget package
* Note: that JavaScripts support is an experimental and does not support complex JavaScripts code.
*/
//IConfiguration config = Configuration.Default.WithDefaultLoader().WithCss().WithCookies().WithJavaScript();
//IBrowsingContext context = BrowsingContext.New(config);
//IDocument document = await context.OpenAsync(url);
//Used PuppeteerSharp loading of HTML document
var htmlString = await TestPuppeteerSharp();
/*
* Parsing of HTML document string
*/
var context = BrowsingContext.New(Configuration.Default);
var parser = context.GetService<IHtmlParser>();
var document = parser.ParseDocument(htmlString);
//Selector carbox element list
var carboxList = document.QuerySelectorAll("div.shop-content div.content div.list li.carbox");
var carModelList = new List<CarModel>();
foreach (var carbox in carboxList)
{
//Parsing and converting to the car model object.
var model = CreateModelWithAngleSharp(carbox);
carModelList.Add(model);
//Printing to console windows
var jsonString = JsonConvert.SerializeObject(model);
Console.WriteLine(jsonString);
Console.WriteLine();
}
Console.WriteLine("Total count:" + carModelList.Count);
}
private static async Task<string> TestPuppeteerSharp()
{
//Enabled headless option
var launchOptions = new LaunchOptions { Headless = true };
//Starting headless browser
var browser = await Puppeteer.LaunchAsync(launchOptions);
//New tab page
var page = await browser.NewPageAsync();
//Request URL to get the page
await page.GoToAsync(Url);
//Get and return the HTML content of the page
var htmlString = await page.GetContentAsync();
#region Dispose resources
//Close tab page
await page.CloseAsync();
//Close headless browser, all pages will be closed here.
await browser.CloseAsync();
#endregion
return htmlString;
}
private static CarModel CreateModelWithAngleSharp(IParentNode node)
{
var model = new CarModel
{
Title = node.QuerySelector("a div.carbox-title").TextContent,
ImageUrl = node.QuerySelector("a div.carbox-carimg img").GetAttribute("src"),
ProductUrl = node.QuerySelector("a").GetAttribute("href"),
Tip = node.QuerySelector("a div.carbox-tip").TextContent,
OrdersNumber = node.QuerySelector("a div.carbox-number span").TextContent
};
return model;
}
}
}
Result
Note
注意，第一次运行，这一句代码：
await new BrowserFetcher().DownloadAsync(ChromiumRevision);
会从网络上下载浏览器便捷式安装包download-Win64-536395.zip到你本地，里面解压后是一个Chromium浏览器。这里需要等待一些时间。