在nutch中，默认情况下尊重robot.txt的配置，同时不提供配置项以忽略robot.txt。
以下是其中一个解释。即作为apache的一个开源项目，必须遵循某些规定，同时由于开放了源代码，可以简单的通过修改源代码来忽略robot.txt的限制。
From the point of view of research and crawling certain pieces of the web, and i strongly agree with you that it should be configurable. But because Nutch being an Apache project, i dismiss it (arguments available upon request). We should adhere to some ethics,
it is bad enough that we can just DoS a server by setting some options to a high level. We publish source code, it leaves the option open to everyone to change it, and i think the current situation is balanced enough.
Patching it is simple, i think we should keep it like that :)
以下为修改源代码的方法：【未验证】
修改类org.apache.nutch.fetcher.FetcherReducer.java
将以下内容注释掉：
if (!rules.isAllowed(fit.u.toString())) {
// unblock
fetchQueues.finishFetchItem(fit, true);
if (LOG.isDebugEnabled()) {
LOG.debug("Denied by robots.txt: " + fit.url);
}
output(fit, null, ProtocolStatusUtils.STATUS_ROBOTS_DENIED,
CrawlStatus.STATUS_GONE);
continue;
}版权声明：本文为博主原创文章，未经博主允许不得转载。