.net 下如何将文档文件(Word, Pdf等) 中的文本提取出来

经常有人问我怎么将类似word，pdf这样的文档转换为文本然后索引，.net 这方面的解决方案不是很多，为了方便大家，我花了一天时间自己做了一个。

Java 版本的 lucence 提供了一个 tika 的工具用于将 word, excel, pdf 等文档转换为文本，然后进行索引。但这个工具没有 .net 版本，要在 .net 下用，需要用 IKVM.net，很麻烦。而且这个工具实际上底层是调用 POI 和 PDFParse 来转换的。从网上搜索到的信息看,POI 对 office 2007 以上版本的文档处理有问题，不知道最新版本是否解决了，我没有试过。 PDFParse 这个东西，我用过 .net 版本，对中文不支持，不知道 Java 版本是否支持。

其实 .net 下完全不需要用这些开源解决方案来解决，因为微软提供了一个官方的解决方案，这个解决方案叫 IFilter，这个过滤器是为 SQL SERVER 的全文索引设计的，但第三方软件可以调用API来完成文档的提取工作。

为了方便大家，我把 IFilter 转换的功能封装到了一个开源的组件中去，大家可以到下面地址去下载源码：HBTextParse .

调用很简单：

这个是提取文件中的文本到字符串的代码

if (openFileDialog.ShowDialog() == DialogResult.OK)
{
    //要转换的文件
    textBoxFilePath.Text = openFileDialog.FileName;
    
    //实例化 TextParse ，传入要转换的文件路径
    TextParse textParse = new TextParse(textBoxFilePath.Text);
    
    //提取文件中的文本，并输出
    richTextBoxView.Text = textParse.ConvertToString();
}

这个是将文件转换为文本文件的代码：

if (saveFileDialog.ShowDialog() == DialogResult.OK)
{
    try
    {
        //实例化 TextParse，传入要转换的文件的路径
        TextParse textParse = new TextParse(textBoxFilePath.Text);
        
        //将文件转换到 saveFileDialog.FileName 指定的文本文件中
        textParse.ConvertToFile(saveFileDialog.FileName);
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
    }
}

要注意的问题是提取 Pdf 文档，如果机器是 64为操作系统，必须要安装 Adobe PDF iFilter 9 for 64-bit platforms. 否则会报异常。这个问题我搞了将近一天才搞定。

支持的文档类型：

目前这个组件可以支持所有 Microsoft office 提供的文档类型，包括 *.rtf, *.doc, *.docx, *.xls, *.xlsx, *.ppt, *.pptx 等等

除了微软Office的文档外，还可以转换

• html 文档:可以把html文档中的文本提取出来（不包含标签）
• Pdf 文档:我测试过，对中文支持没有问题
• Txt 文档

这个代码的核心部分是一个叫 FilterCode 的类。这个类是从 http://ifilter.codeplex.com/ 这个地方下载的，我对这个类做了改进，加入了转换到文件的方法。我把这个类的代码贴出来，如果对如何调用IFilter的windows API感兴趣，可以参考这段代码

IFilter 的相关API函数如下：通常这些API函数就可以通过IFilter接口提取文本。

        [DllImport("query.dll", SetLastError = true, CharSet = CharSet.Unicode)]
        static extern int LoadIFilter(string pwcsPath,
                  [MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter,
                  ref IFilter ppIUnk);

        [ComImport, Guid("89BCB740-6119-101A-BCB7-00DD010655AF")]
        [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
        public interface IFilter
        {
            /// <summary>
            /// The IFilter::Init method initializes a filtering session.
            /// </summary>
            [PreserveSig]
            IFilterReturnCodes Init(
                //[in] Flag settings from the IFILTER_INIT enumeration for
                // controlling text standardization, property output, embedding
                // scope, and IFilter access patterns. 
              IFILTER_INIT grfFlags,

              // [in] The size of the attributes array. When nonzero, cAttributes
                //  takes 
                // precedence over attributes specified in grfFlags. If no
                // attribute flags 
                // are specified and cAttributes is zero, the default is given by
                // the 
                // PSGUID_STORAGE storage property set, which contains the date and
                //  time 
                // of the last write to the file, size, and so on; and by the
                //  PID_STG_CONTENTS 
                // 'contents' property, which maps to the main contents of the
                // file. 
                // For more information about properties and property sets, see
                // Property Sets. 
              int cAttributes,

              //[in] Array of pointers to FULLPROPSPEC structures for the
                // requested properties. 
                // When cAttributes is nonzero, only the properties in aAttributes
                // are returned. 
              IntPtr aAttributes,

              // [out] Information about additional properties available to the
                //  caller; from the IFILTER_FLAGS enumeration. 
              out IFILTER_FLAGS pdwFlags);

            /// <summary>
            /// The IFilter::GetChunk method positions the filter at the beginning
            /// of the next chunk, 
            /// or at the first chunk if this is the first call to the GetChunk
            /// method, and returns a description of the current chunk. 
            /// </summary>
            [PreserveSig]
            IFilterReturnCodes GetChunk(out STAT_CHUNK pStat);

            /// <summary>
            /// The IFilter::GetText method retrieves text (text-type properties)
            /// from the current chunk, 
            /// which must have a CHUNKSTATE enumeration value of CHUNK_TEXT.
            /// </summary>
            [PreserveSig]
            IFilterReturnCodes GetText(
                // [in/out] On entry, the size of awcBuffer array in wide/Unicode
                // characters. On exit, the number of Unicode characters written to
                // awcBuffer. 
                // Note that this value is not the number of bytes in the buffer. 
                ref int pcwcBuffer,

                // Text retrieved from the current chunk. Do not terminate the
                // buffer with a character.  
                [Out(), MarshalAs(UnmanagedType.LPWStr)] 
       StringBuilder awcBuffer);

            /// <summary>
            /// The IFilter::GetValue method retrieves a value (public
            /// value-type property) from a chunk, 
            /// which must have a CHUNKSTATE enumeration value of CHUNK_VALUE.
            /// </summary>
            [PreserveSig]
            IFilterReturnCodes GetValue(
                // Allocate the PROPVARIANT structure with CoTaskMemAlloc. Some
                // PROPVARIANT 
                // structures contain pointers, which can be freed by calling the
                // PropVariantClear function. 
                // It is up to the caller of the GetValue method to call the
                //   PropVariantClear method.            
                // ref IntPtr ppPropValue
                // [MarshalAs(UnmanagedType.Struct)]
                ref IntPtr PropVal);

            /// <summary>
            /// The IFilter::BindRegion method retrieves an interface representing
            /// the specified portion of the object. 
            /// Currently reserved for future use.
            /// </summary>
            [PreserveSig]
            IFilterReturnCodes BindRegion(ref FILTERREGION origPos,
              ref Guid riid, ref object ppunk);
        }

从文档中提取文本的代码如下：

        /// <summary>
        /// Utilizes IFilter interface in Windows to parse the contents of files.
        /// </summary>
        /// <param name="path">Path - Location of file to parse</param>
        /// <param name="buffer">Buffer - Return text artifacts</param>
        /// <returns>Raw set of strings from the document in plain text format.</returns>
        public void GetTextFromDocument(string path, ref StringBuilder buffer)
        {
            IFilter filter = null;
            int hresult;
            IFilterReturnCodes rtn;

            // Initialize the return buffer to 64K.
            buffer = new StringBuilder(64 * 1024);

            // Try to load the filter for the path given.
            hresult = LoadIFilter(path, new IntPtr(0), ref filter);
            if (hresult == 0)
            {
                IFILTER_FLAGS uflags;

                // Init the filter provider.
                rtn = filter.Init(
                        IFILTER_INIT.IFILTER_INIT_CANON_PARAGRAPHS |
                        IFILTER_INIT.IFILTER_INIT_CANON_HYPHENS |
                        IFILTER_INIT.IFILTER_INIT_CANON_SPACES |
                        IFILTER_INIT.IFILTER_INIT_APPLY_INDEX_ATTRIBUTES |
                        IFILTER_INIT.IFILTER_INIT_INDEXING_ONLY,
                        0, new IntPtr(0), out uflags);
                if (rtn == IFilterReturnCodes.S_OK)
                {
                    STAT_CHUNK statChunk;

                    // Outer loop will read chunks from the document at a time.  For those
                    // chunks that have text, the contents will be pulled and put into the
                    // return buffer.
                    bool bMoreChunks = true;
                    while (bMoreChunks)
                    {
                        rtn = filter.GetChunk(out statChunk);
                        if (rtn == IFilterReturnCodes.S_OK)
                        {
                            // Ignore all non-text chunks.
                            if (statChunk.flags != CHUNKSTATE.CHUNK_TEXT)
                                continue;

                            // Check for white space items and add the appropriate breaks.
                            switch (statChunk.breakType)
                            {
                                case CHUNK_BREAKTYPE.CHUNK_NO_BREAK:
                                    break;

                                case CHUNK_BREAKTYPE.CHUNK_EOW:
                                    buffer.Append(' ');
                                    break;

                                case CHUNK_BREAKTYPE.CHUNK_EOC:
                                case CHUNK_BREAKTYPE.CHUNK_EOP:
                                case CHUNK_BREAKTYPE.CHUNK_EOS:
                                    buffer.AppendLine();
                                    break;
                            }

                            // At this point we have a text chunk.  The following code will pull out
                            // all of it and add it to the buffer.
                            bool bMoreText = true;
                            while (bMoreText)
                            {
                                // Create a temporary string buffer we can use for the parsing algorithm.
                                int cBuffer = DefaultBufferSize;
                                StringBuilder sbBuffer = new StringBuilder(DefaultBufferSize);

                                // Read the next piece of data up to the size of our local buffer.
                                rtn = filter.GetText(ref cBuffer, sbBuffer);
                                if (rtn == IFilterReturnCodes.S_OK || rtn == IFilterReturnCodes.FILTER_S_LAST_TEXT)
                                {
                                    // If any data was returned, scrub it and then add it to the buffer.
                                    CleanUpCharacters(cBuffer, sbBuffer);
                                    buffer.Append(sbBuffer.ToString());

                                    // If we got back some text but there is no more, terminate the loop.
                                    if (rtn == IFilterReturnCodes.FILTER_S_LAST_TEXT)
                                    {
                                        bMoreText = false;
                                        break;
                                    }
                                }
                                // Once all data is exhausted, we are done so terminate.
                                else if (rtn == IFilterReturnCodes.FILTER_E_NO_MORE_TEXT)
                                {
                                    bMoreText = false;
                                    break;
                                }
                                // Check for any fatal errors.  It is a bug if you land here.
                                else if (rtn == IFilterReturnCodes.FILTER_E_NO_TEXT)
                                {
                                    System.Diagnostics.Debug.Assert(false, "Should not get here");
                                    throw new InvalidOperationException();
                                }
                            }
                        }
                        // Once all chunks have been read, we are done with the file.
                        else if (rtn == IFilterReturnCodes.FILTER_E_END_OF_CHUNKS)
                        {
                            bMoreChunks = false;
                            break;
                        }
                        else if (rtn == IFilterReturnCodes.FILTER_E_EMBEDDING_UNAVAILABLE ||
                            rtn == IFilterReturnCodes.FILTER_E_LINK_UNAVAILABLE)
                        {
                            continue;
                        }
                        else
                        {
                            throw new COMException("IFilter COM error: " + rtn.ToString());
                        }
                    }
                }
            }
            else
            {
                // If you get here there is no filter for the file type you asked for.  Throw an
                // exception for the caller.
                throw new InvalidOperationException("Failed to find IFilter for file " + path);
            }
        }

 

 

分类: Hubble.Net, 搜索引擎

标签: Hubble.Net, 索引