Scrapy 如何控制导出顺序

Scrapy 如何控制导出顺序

1. 遇到的问题

在用Scrapy到处item的时候，发现顺序错乱（应该是按照abc的顺序排列的），并不是items.py文件中定义的顺序，那么如何控制呢？

2. fields_to_export

我在查看官网文档的时候找到了这个属性，它的解释是这样的：

fields_to_export

A list with the name of the fields that will be exported, or None if you want to export all fields. Defaults to None.

Some exporters (like CsvItemExporter) respect the order of the fields defined in this attribute.

When using item objects that do not expose all their possible fields, exporters that do not support exporting a different subset of fields per item will only export the fields found in the first item exported. Use fields_to_export to define all the fields to be exported.

大致意思是：这个列表（它是一个列表）可以控制导出的字段个数，但是在一些导出器像CsvItemExporter可以控制导出字段的顺序

所以：只需要在使用Exporter的时候，传一个fields_to_export的参数，就可以控制导出字段的个数/顺序

3. 示例

pipelines.py

from scrapy.exporters import JsonLinesItemExporter, CsvItemExporter
from itemadapter import ItemAdapter

fields_to_export = ['city_name', 'house_addr', 'house_class', 'house_size', 'house_facility', 'house_price',
                    'house_release_time']


class JsonLinesItemPipeline:

    def __init__(self):
        self.file = open('storages/renting.jl', 'wb')
        self.exporter = JsonLinesItemExporter(self.file, encoding='utf-8', fields_to_export=fields_to_export)

    def open_spider(self, spider):
        pass

    def process_item(self, item, spider):
        self.exporter.export_item(item)
        return item

    def close_spider(self, spider):
        self.file.close()


class CsvItemPipeline:

    def __init__(self):
        self.file = open('storages/renting.csv', 'wb')
        self.exporter = CsvItemExporter(self.file, fields_to_export=fields_to_export)

    def open_spider(self, spider):
        pass

    def process_item(self, item, spider):
        self.exporter.export_item(item)
        return item

    def close_spider(self, spider):
        self.file.close()

---

参考：

• https://docs.scrapy.org/en/latest/topics/exporters.html#scrapy.exporters.BaseItemExporter.fields_to_export
• https://docs.scrapy.org/en/latest/topics/exporters.html#csvitemexporter