Lucene排序 Payload的应用

有关Lucene的Payload的相关内容，可以参考如下链接，介绍的非常详细，值得参考：

http://www.ibm.com/developerworks/cn/opensource/os-cn-lucene-pl/
http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/

例如，有这样的一个需求：

现在有两篇文档内容非常相似，如下所示：

文档1：egg tomato potato bread  
文档2：egg book potato bread  

现在我想要查询食物（foods），而且是查询关键词是egg，如何能够区别出上面两个文档哪一个更是我想要的？

可以看到上面两篇文档，文档1中描述的各项都是食物，而文档2中的book不是食物，基于上述需求，应该是文档1比文档2更相关，在查询结果中，文档1排名应该更靠前。通过上面
http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/中给出的方法，可以在文档中，对给定词出现在文档的出现的权重信息（egg在文档1与文档中，以foods来衡量，文档1更相关），可以在索引之前处理一下，为egg增加payload信息，例如：

 

文档1：egg|0.984 tomato potato bread  
文档2：egg|0.356 book potato bread  

然后再进行索引，通过Lucene提供的PayloadTermQuery就能够分辨出上述egg这个Term的不同。在Lucene中，实际上是将我们存储的Payload数据，如上述"|"分隔后面的数字，乘到了tf上，然后在进行权重的计算。

下面，我们再看一下，增加一个Field来存储Payload数据，而源文档不需要进行修改，或者，我们可以在索引之前对文档进行一个处理，例如分类，通过分类可以给不同的文档所属类别的不同程度，计算一个Payload数值。

为了能够使用存储的Payload数据信息，结合上面提出的实例，我们需要按照如下步骤去做：

第一，待索引数据处理

例如，增加category这个Field存储类别信息，content这个Field存储上面的内容：

文档1：  
new Field("category", "foods|0.984 shopping|0.503", Field.Store.YES, Field.Index.ANALYZED)  
new Field("content", "egg tomato potato bread", Field.Store.YES, Field.Index.ANALYZED)  
文档2：  
new Field("category", "foods|0.356 shopping|0.791", Field.Store.YES, Field.Index.ANALYZED)  
new Field("content", "egg book potato bread", Field.Store.YES, Field.Index.ANALYZED)  

   

第二，实现解析Payload数据的Analyzer

由于Payload信息存储在category这个Field中，多个类别之间使用空格分隔，每个类别内容是以"|"分隔的，所以我们的Analyzer就要能够解析它。Lucene提供了DelimitedPayloadTokenFilter，能够处理具有分隔符的情况。我们的实现如下所示：

package org.shirdrn.lucene.query.payloadquery;  
    
import java.io.Reader;  
    
import org.apache.lucene.analysis.Analyzer;  
import org.apache.lucene.analysis.TokenStream;  
import org.apache.lucene.analysis.WhitespaceTokenizer;  
import org.apache.lucene.analysis.payloads.DelimitedPayloadTokenFilter;  
import org.apache.lucene.analysis.payloads.PayloadEncoder;  
    
public class PayloadAnalyzer extends Analyzer {  
    private PayloadEncoder encoder;  
    
    PayloadAnalyzer(PayloadEncoder encoder) {  
        this.encoder = encoder;  
    }  
    
    @SuppressWarnings("deprecation")  
    public TokenStream tokenStream(String fieldName, Reader reader) {  
        TokenStream result = new WhitespaceTokenizer(reader); // 用来解析空格分隔的各个类别  
        result = new DelimitedPayloadTokenFilter(result, '|', encoder); // 在上面分词的基础上，在进行Payload数据解析  
        return result;  
    }  
}  

第三， 实现Similarity计算得分

Lucene中Similarity类中提供了scorePayload方法，用于计算Payload值来对文档贡献得分，我们重写了该方法，实现如下所示：

package org.shirdrn.lucene.query.payloadquery;  
    
import org.apache.lucene.analysis.payloads.PayloadHelper;  
import org.apache.lucene.search.DefaultSimilarity;  
    
    
public class PayloadSimilarity extends DefaultSimilarity {  
    
    private static final long serialVersionUID = 1L;  
    
    @Override  
    public float scorePayload(int docId, String fieldName, int start, int end,  
            byte[] payload, int offset, int length) {  
        return PayloadHelper.decodeFloat(payload, offset);  
    }  
    
}  

通过使用PayloadHelper这个工具类可以获取到Payload值，然后在计算文档得分的时候起到作用。

第四，创建索引

在创建索引的时候，需要使用到我们上面实现的Analyzer和Similarity，代码如下所示：

package org.shirdrn.lucene.query.payloadquery;  
    
import java.io.File;  
import java.io.IOException;  
    
import org.apache.lucene.analysis.Analyzer;  
import org.apache.lucene.analysis.payloads.FloatEncoder;  
import org.apache.lucene.document.Document;  
import org.apache.lucene.document.Field;  
import org.apache.lucene.index.CorruptIndexException;  
import org.apache.lucene.index.IndexWriter;  
import org.apache.lucene.index.IndexWriterConfig;  
import org.apache.lucene.index.IndexWriterConfig.OpenMode;  
import org.apache.lucene.search.Similarity;  
import org.apache.lucene.store.FSDirectory;  
import org.apache.lucene.store.LockObtainFailedException;  
import org.apache.lucene.util.Version;  
    
public class PayloadIndexing {  
    
    private IndexWriter indexWriter = null;  
    private final Analyzer analyzer = new PayloadAnalyzer(new FloatEncoder()); // 使用PayloadAnalyzer，并指定Encoder  
    private final Similarity similarity = new PayloadSimilarity(); // 实例化一个PayloadSimilarity  
    private IndexWriterConfig config = null;  
        
    public PayloadIndexing(String indexPath) throws CorruptIndexException, LockObtainFailedException, IOException {  
        File indexFile = new File(indexPath);  
        config = new IndexWriterConfig(Version.LUCENE_31, analyzer);  
        config.setOpenMode(OpenMode.CREATE).setSimilarity(similarity); // 设置计算得分的Similarity  
        indexWriter = new IndexWriter(FSDirectory.open(indexFile), config);  
    }  
    
    public void index() throws CorruptIndexException, IOException {       
        Document doc1 = new Document();  
        doc1.add(new Field("category", "foods|0.984 shopping|0.503", Field.Store.YES, Field.Index.ANALYZED));  
        doc1.add(new Field("content", "egg tomato potato bread", Field.Store.YES, Field.Index.ANALYZED));  
        indexWriter.addDocument(doc1);  
            
        Document doc2 = new Document();  
        doc2.add(new Field("category", "foods|0.356 shopping|0.791", Field.Store.YES, Field.Index.ANALYZED));  
        doc2.add(new Field("content", "egg book potato bread", Field.Store.YES, Field.Index.ANALYZED));  
        indexWriter.addDocument(doc2);  
            
        indexWriter.close();  
    }  
        
    public static void main(String[] args) throws CorruptIndexException, IOException {  
        new PayloadIndexing("E:\\index").index();  
    }  
}  

第五，查询

查询的时候，我们可以构造PayloadTermQuery来进行查询。代码如下所示：

 

package org.shirdrn.lucene.query.payloadquery;  
    
import java.io.File;  
import java.io.IOException;  
    
import org.apache.lucene.document.Document;  
import org.apache.lucene.index.CorruptIndexException;  
import org.apache.lucene.index.IndexReader;  
import org.apache.lucene.index.Term;  
import org.apache.lucene.queryParser.ParseException;  
import org.apache.lucene.search.BooleanQuery;  
import org.apache.lucene.search.Explanation;  
import org.apache.lucene.search.IndexSearcher;  
import org.apache.lucene.search.ScoreDoc;  
import org.apache.lucene.search.TopScoreDocCollector;  
import org.apache.lucene.search.BooleanClause.Occur;  
import org.apache.lucene.search.payloads.AveragePayloadFunction;  
import org.apache.lucene.search.payloads.PayloadTermQuery;  
import org.apache.lucene.store.NIOFSDirectory;  
    
public class PayloadSearching {  
        
    private IndexReader indexReader;  
    private IndexSearcher searcher;  
        
    public PayloadSearching(String indexPath) throws CorruptIndexException, IOException {  
        indexReader = IndexReader.open(NIOFSDirectory.open(new File(indexPath)), true);  
        searcher = new IndexSearcher(indexReader);  
        searcher.setSimilarity(new PayloadSimilarity()); // 设置自定义的PayloadSimilarity  
    }  
        
    public ScoreDoc[] search(String qsr) throws ParseException, IOException {  
        int hitsPerPage = 10;  
        BooleanQuery bq = new BooleanQuery();  
        for(String q : qsr.split(" ")) {  
            bq.add(createPayloadTermQuery(q), Occur.MUST);  
        }  
        TopScoreDocCollector collector = TopScoreDocCollector.create(5 * hitsPerPage, true);  
        searcher.search(bq, collector);  
        ScoreDoc[] hits = collector.topDocs().scoreDocs;  
        for (int i = 0; i < hits.length; i++) {  
            int docId = hits[i].doc; // 文档编号  
            Explanation  explanation  = searcher.explain(bq, docId);  
            System.out.println(explanation.toString());  
        }  
        return hits;  
    }  
        
    public void display(ScoreDoc[] hits, int start, int end) throws CorruptIndexException, IOException {  
        end = Math.min(hits.length, end);  
        for (int i = start; i < end; i++) {  
            Document doc = searcher.doc(hits[i].doc);  
            int docId = hits[i].doc; // 文档编号  
            float score = hits[i].score; // 文档得分  
            System.out.println(docId + "\t" + score + "\t" + doc + "\t");  
        }  
    }  
        
    public void close() throws IOException {  
        searcher.close();  
        indexReader.close();  
    }  
        
    private PayloadTermQuery createPayloadTermQuery(String item) {  
        PayloadTermQuery ptq = null;  
        if(item.indexOf("^")!=-1) {  
            String[] a = item.split("\\^");  
            String field = a[0].split(":")[0];  
            String token = a[0].split(":")[1];  
            ptq = new PayloadTermQuery(new Term(field, token), new AveragePayloadFunction());  
            ptq.setBoost(Float.parseFloat(a[1].trim()));  
        } else {  
            String field = item.split(":")[0];  
            String token = item.split(":")[1];  
            ptq = new PayloadTermQuery(new Term(field, token), new AveragePayloadFunction());  
        }  
        return ptq;  
    }  
        
    public static void main(String[] args) throws ParseException, IOException {  
        int start = 0, end = 10;      
//      String queries = "category:foods^123.0 content:bread^987.0";  
        String queries = "category:foods content:egg";  
        PayloadSearching payloadSearcher = new PayloadSearching("E:\\index");  
        payloadSearcher.display(payloadSearcher.search(queries), start, end);  
        payloadSearcher.close();  
    }  
    
}  

我们可以看到查询结果，两个文档的相关度排序：

0   0.3314532   Document<stored,indexed,tokenized<category:foods|0.984 shopping|0.503> stored,indexed,tokenized<content:egg tomato potato bread>>   
1   0.21477573  Document<stored,indexed,tokenized<category:foods|0.356 shopping|0.791> stored,indexed,tokenized<content:egg book potato bread>>  

通过输出计算得分的解释信息，如下所示：

1. 0.3314532 = (MATCH) sum of:  
2.   0.18281947 = (MATCH) weight(category:foods in 0), product of:  
3.     0.70710677 = queryWeight(category:foods), product of:  
4.       0.5945349 = idf(category:  foods=2)  
5.       1.1893445 = queryNorm  
6.     0.2585458 = (MATCH) fieldWeight(category:foods in 0), product of:  
7.       0.6957931 = (MATCH) btq, product of:  
8.         0.70710677 = tf(phraseFreq=0.5)  
9.         0.984 = scorePayload(...)  
10.       0.5945349 = idf(category:  foods=2)  
11.       0.625 = fieldNorm(field=category, doc=0)  
12.   0.14863372 = (MATCH) weight(content:egg in 0), product of:  
13.     0.70710677 = queryWeight(content:egg), product of:  
14.       0.5945349 = idf(content:  egg=2)  
15.       1.1893445 = queryNorm  
16.     0.21019982 = (MATCH) fieldWeight(content:egg in 0), product of:  
17.       0.70710677 = (MATCH) btq, product of:  
18.         0.70710677 = tf(phraseFreq=0.5)  
19.         1.0 = scorePayload(...)  
20.       0.5945349 = idf(content:  egg=2)  
21.       0.5 = fieldNorm(field=content, doc=0)  
22.     
23. 0.21477571 = (MATCH) sum of:  
24.   0.066142 = (MATCH) weight(category:foods in 1), product of:  
25.     0.70710677 = queryWeight(category:foods), product of:  
26.       0.5945349 = idf(category:  foods=2)  
27.       1.1893445 = queryNorm  
28.     0.09353892 = (MATCH) fieldWeight(category:foods in 1), product of:  
29.       0.25173002 = (MATCH) btq, product of:  
30.         0.70710677 = tf(phraseFreq=0.5)  
31.         0.356 = scorePayload(...)  
32.       0.5945349 = idf(category:  foods=2)  
33.       0.625 = fieldNorm(field=category, doc=1)  
34.   0.14863372 = (MATCH) weight(content:egg in 1), product of:  
35.     0.70710677 = queryWeight(content:egg), product of:  
36.       0.5945349 = idf(content:  egg=2)  
37.       1.1893445 = queryNorm  
38.     0.21019982 = (MATCH) fieldWeight(content:egg in 1), product of:  
39.       0.70710677 = (MATCH) btq, product of:  
40.         0.70710677 = tf(phraseFreq=0.5)  
41.         1.0 = scorePayload(...)  
42.       0.5945349 = idf(content:  egg=2)  
43.       0.5 = fieldNorm(field=content, doc=1)  

我们可以看到，除了在tf上乘了一个Payload值以外，其他的都相同，也就是说，我们预期使用的Payload为文档（ID=0）贡献了得分，排名靠前了。否则，如果不使用Payload的话，查询结果中两个文档的得分是相同的（可以模拟设置他们的Payload值相同，测试一下看看）

    

---

相关文章阅读及免费下载：

《Lucene Ranking算法分析》

《Lucene Payload 的研究与应用》

《Lucene排序 Payload的应用》

《Apache Lucene3.0结果排序原理 操作 示例》

更多《Apache Lucene文档》，尽在开卷有益360  http://www.docin.com/book_360