Nutch 使用入门(一）——准备工作及Intranet抓取
http://softkid.iteye.com/blog/625736http://51mst.iteye.com/blog/1155120
nutch1.3 +hadoop 分布式部署(亲测)
1.确保hadoop正常启动
2.下载nutch1.3 安装包 解压到指定路径
3.抓取
nutch1.3 有两个conf 一个在NUTCH_HOME/conf ,另一个在rumtime/local/conf
runtime/local/conf 为 local(本地抓取的配置文件所用)
NUTCH_HOME/conf 为分布式抓取所用
下面我们着重讲解 分布式抓取
4.分布式抓取：
rutime/deply/bin/nutch下执行分布式抓取命令(分布式抓取一定是在这个下面，local为本地抓取所用)
chmod +x bin/nutch 赋予执行权限
5.拷贝hadoop环境
将HADOOP_HOME/conf下的 6个文件：
core-site.xml
hadoop-env.sh
hdfs-site.xml
mapred-site.xml
masters
slaves
拷贝到NUTCH_HOME/conf下
6.配置nutch-site.xml
简单配置一个http.agent.name 即可
<property>
<name>http.agent.name</name>
<value>MyCrawl001</value>
</property>
7.配置regex-urlfilter.txt抓取动态网页
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
+[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
+.
8.native/lib
这是配置nutch1.3 hadoop集群最重要一点
下面是NUTCH_HOME/lib/native下的 文档README.txt
These libraries are purely optional, and if they are missing Hadoop will
use corresponding pure Java components. The impact of native compression
becomes noticeable with larger datasets and weaker CPU-s - if you notice
that the CPU is routinely saturated when a job is sorting or reducing,
then using these libs may help.
Installation instructions
=========================
You can obtain the necessary files from a distribution package of Hadoop,
e.g. hadoop-0.20.2.tar.gz. Unpack this archive, and copy the content of
lib/native here, so that the layout looks like this:
<Nutch home>/lib/native/Linux-amd64-64/...
<Nutch home>/lib/native/Linux-i386-32/...
Local runtime
-------------
The build process will include these native libraries when preparing
the /runtime/local environment for running in local mode.
/runtime/local/bin/nutch knows how to use these libs - if they are
found and correctly used you should see lines like this in your logs:
Distributed runtime
-------------------
If you want to use this component in an existing Hadoop cluster (when using
/runtime/deploy artifacts) you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster. If
you installed the cluster from a distribution package of Hadoop then these
libraries should already be in the right place and you shouldn't need to do
anything else.
~
大体意思就是说 可以将 HADOOP_HOME下的 lib/native中的文件
Linux-amd64-64
Linux-i386-32
拷贝到NUTCH_HOME/lib/native下(按英文原意要确保you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster 确保这个文件在每一个节点上 并且重启集群,我拷贝了,)
9.执行
runtime/deply/bin/nutch hdfs://server0:9000/user/suse/urls -dir crawl -depth 200 -threads 200 -topN 1000
10.成功
看到map-reduce 任务成功执行 则配置成功
11/08/22 16:33:26 INFO mapred.JobClient: Reduce input records=48148
11/08/22 16:33:26 INFO crawl.CrawlDb: CrawlDb update: finished at 2011-08-22 16:33:26, elapsed: 00:00:39
11/08/22 16:33:26 INFO crawl.Generator: Generator: starting at 2011-08-22 16:33:26
11/08/22 16:33:26 INFO crawl.Generator: Generator: Selecting best-scoring urls due for fetch.
11/08/22 16:33:26 INFO crawl.Generator: Generator: filtering: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: normalizing: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: topN: 1000
11/08/22 16:33:27 INFO mapred.FileInputFormat: Total input paths to process : 8
11/08/22 16:33:28 INFO mapred.JobClient: Running job: job_201108221601_0022
11/08/22 16:33:29 INFO mapred.JobClient: map 0% reduce 0%
11/08/22 16:33:36 INFO mapred.JobClient: map 25% reduce 0%
11/08/22 16:33:39 INFO mapred.JobClient: map 50% reduce 0%
11/08/22 16:33:42 INFO mapred.JobClient: map 75% reduce 0%
11/08/22 16:33:45 INFO mapred.JobClient: map 75% reduce 4%
11/08/22 16:33:46 INFO mapred.JobClient: map 100% reduce 4%
11/08/22 16:33:48 INFO mapred.JobClient: map 100% reduce 13%
11/08/22 16:33:49 INFO mapred.JobClient: map 100% reduce 19%
11/08/22 16:33:51 INFO mapred.JobClient: map 100% reduce 33%
11/08/22 16:33:54 INFO mapred.JobClient: map 100% reduce 53%
11/08/22 16:33:57 INFO mapred.JobClient: map 100% reduce 71%
11/08/22 16:33:58 INFO mapred.JobClient: map 100% reduce 90%
11/08/22 16:34:00 INFO mapred.JobClient: map 100% reduce 100%
11/08/22 16:34:02 INFO mapred.JobClient: Job complete: job_201108221601_0022
11.solr 索引
nutch1.3下只有三个文件夹
crawldb
linkdb
segments
必须要用solr去建立索引
12.遇到错误
question：
Exception in thread "main" java.lang.IllegalArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1166)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1068)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
answer:
using 1.3? If so make sure you changed nutch-site.xml (and not default)
in runtime/local/conf Changing the conf in NUTCH_HOME/conf won't be copied
to the runtime dirs unless you rebuild with ant.
BTW why don't you ask on the mailing list instead? You are more likely to get some help there
这个是英文元版回答。主要意思就是 nutch1.3 修改完nutch-site.xml 要重新ant一下
question：
Caused by: java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:61)
at java.io.BufferedReader.<init>(BufferedReader.java:76)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.readRules(RegexURLFilterBase.java:180)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.setConf(RegexURLFilterBase.java:156)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:162)
at org.apache.nutch.net.URLFilters.<init>(URLFilters.java:57)
at org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:72)
... 18 more
answer：
在runtime/local/下执行分布式抓取则报这个错误
注意：分布式抓取一定是在runtime/deply下执行
bin/nutch 命令
1.确保hadoop正常启动
2.下载nutch1.3 安装包 解压到指定路径
3.抓取
nutch1.3 有两个conf 一个在NUTCH_HOME/conf ,另一个在rumtime/local/conf
runtime/local/conf 为 local(本地抓取的配置文件所用)
NUTCH_HOME/conf 为分布式抓取所用
下面我们着重讲解 分布式抓取
4.分布式抓取：
rutime/deply/bin/nutch下执行分布式抓取命令(分布式抓取一定是在这个下面，local为本地抓取所用)
chmod +x bin/nutch 赋予执行权限
5.拷贝hadoop环境
将HADOOP_HOME/conf下的 6个文件：
core-site.xml
hadoop-env.sh
hdfs-site.xml
mapred-site.xml
masters
slaves
拷贝到NUTCH_HOME/conf下
6.配置nutch-site.xml
简单配置一个http.agent.name 即可
<property>
<name>http.agent.name</name>
<value>MyCrawl001</value>
</property>
7.配置regex-urlfilter.txt抓取动态网页
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
+[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
+.
8.native/lib
这是配置nutch1.3 hadoop集群最重要一点
下面是NUTCH_HOME/lib/native下的 文档README.txt
These libraries are purely optional, and if they are missing Hadoop will
use corresponding pure Java components. The impact of native compression
becomes noticeable with larger datasets and weaker CPU-s - if you notice
that the CPU is routinely saturated when a job is sorting or reducing,
then using these libs may help.
Installation instructions
=========================
You can obtain the necessary files from a distribution package of Hadoop,
e.g. hadoop-0.20.2.tar.gz. Unpack this archive, and copy the content of
lib/native here, so that the layout looks like this:
<Nutch home>/lib/native/Linux-amd64-64/...
<Nutch home>/lib/native/Linux-i386-32/...
Local runtime
-------------
The build process will include these native libraries when preparing
the /runtime/local environment for running in local mode.
/runtime/local/bin/nutch knows how to use these libs - if they are
found and correctly used you should see lines like this in your logs:
Distributed runtime
-------------------
If you want to use this component in an existing Hadoop cluster (when using
/runtime/deploy artifacts) you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster. If
you installed the cluster from a distribution package of Hadoop then these
libraries should already be in the right place and you shouldn't need to do
anything else.
~
大体意思就是说 可以将 HADOOP_HOME下的 lib/native中的文件
Linux-amd64-64
Linux-i386-32
拷贝到NUTCH_HOME/lib/native下(按英文原意要确保you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster 确保这个文件在每一个节点上 并且重启集群,我拷贝了,)
9.执行
runtime/deply/bin/nutch hdfs://server0:9000/user/suse/urls -dir crawl -depth 200 -threads 200 -topN 1000
10.成功
看到map-reduce 任务成功执行 则配置成功
11/08/22 16:33:26 INFO mapred.JobClient: Reduce input records=48148
11/08/22 16:33:26 INFO crawl.CrawlDb: CrawlDb update: finished at 2011-08-22 16:33:26, elapsed: 00:00:39
11/08/22 16:33:26 INFO crawl.Generator: Generator: starting at 2011-08-22 16:33:26
11/08/22 16:33:26 INFO crawl.Generator: Generator: Selecting best-scoring urls due for fetch.
11/08/22 16:33:26 INFO crawl.Generator: Generator: filtering: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: normalizing: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: topN: 1000
11/08/22 16:33:27 INFO mapred.FileInputFormat: Total input paths to process : 8
11/08/22 16:33:28 INFO mapred.JobClient: Running job: job_201108221601_0022
11/08/22 16:33:29 INFO mapred.JobClient: map 0% reduce 0%
11/08/22 16:33:36 INFO mapred.JobClient: map 25% reduce 0%
11/08/22 16:33:39 INFO mapred.JobClient: map 50% reduce 0%
11/08/22 16:33:42 INFO mapred.JobClient: map 75% reduce 0%
11/08/22 16:33:45 INFO mapred.JobClient: map 75% reduce 4%
11/08/22 16:33:46 INFO mapred.JobClient: map 100% reduce 4%
11/08/22 16:33:48 INFO mapred.JobClient: map 100% reduce 13%
11/08/22 16:33:49 INFO mapred.JobClient: map 100% reduce 19%
11/08/22 16:33:51 INFO mapred.JobClient: map 100% reduce 33%
11/08/22 16:33:54 INFO mapred.JobClient: map 100% reduce 53%
11/08/22 16:33:57 INFO mapred.JobClient: map 100% reduce 71%
11/08/22 16:33:58 INFO mapred.JobClient: map 100% reduce 90%
11/08/22 16:34:00 INFO mapred.JobClient: map 100% reduce 100%
11/08/22 16:34:02 INFO mapred.JobClient: Job complete: job_201108221601_0022
11.solr 索引
nutch1.3下只有三个文件夹
crawldb
linkdb
segments
必须要用solr去建立索引
12.遇到错误
question：
Exception in thread "main" java.lang.IllegalArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1166)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1068)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
answer:
using 1.3? If so make sure you changed nutch-site.xml (and not default)
in runtime/local/conf Changing the conf in NUTCH_HOME/conf won't be copied
to the runtime dirs unless you rebuild with ant.
BTW why don't you ask on the mailing list instead? You are more likely to get some help there
这个是英文元版回答。主要意思就是 nutch1.3 修改完nutch-site.xml 要重新ant一下
question：
Caused by: java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:61)
at java.io.BufferedReader.<init>(BufferedReader.java:76)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.readRules(RegexURLFilterBase.java:180)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.setConf(RegexURLFilterBase.java:156)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:162)
at org.apache.nutch.net.URLFilters.<init>(URLFilters.java:57)
at org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:72)
... 18 more
answer：
在runtime/local/下执行分布式抓取则报这个错误
注意：分布式抓取一定是在runtime/deply下执行
bin/nutch 命令
