如何以正确的顺序导入Scrapy项目密钥? [英] How to import Scrapy item keys in the correct order?

问题描述

我正在将items.py中的Scrapy项目键导入到pipelines.py中. 问题在于，导入项目的顺序与items.py文件中的定义不同.

I am importing the Scrapy item keys from items.py, into pipelines.py. The problem is that the order of the imported items are different from how they were defined in the items.py file.

我的items.py文件:

class NewAdsItem(Item):
    AdId        = Field()
    DateR       = Field()
    AdURL       = Field()

在我的pipelines.py中:

from adbot.items import NewAdsItem
...
def open_spider(self, spider):
     self.ikeys = NewAdsItem.fields.keys()
     print("Keys in pipelines: \t%s" % ",".join(self.ikeys) )
     #self.createDbTable(ikeys)

输出为:

Keys in pipelines:  AdId,AdURL,DateR

而不是预期的:AdId,DateR,AdURL.

如何确保导入的订单保持不变?

注意::这可能与

Note: This might be related to How to get order of fields in Scrapy item, but it's not at all very clear what's going on, since Python3 docs state that lists and dicts should retain their order. Also note, that when using process_item() and using item.keys(), the order is retained! But I need to access the keys in order before item's are scraped.

推荐答案

我唯一可以使它起作用的方法是使用该解决方案以下列方式.

The only way I could get this to work, was to use this solution in the following manner.

我的 items.py 文件:

My items.py file:

from scrapy.item import Item, Field
from collections import OrderedDict
from types import FunctionType

class StaticOrderHelper(type):
    # Requires Python3
    def __prepare__(name, bases, **kwargs):
        return OrderedDict()

    def __new__(mcls, name, bases, namespace, **kwargs):
        namespace['_field_order'] = [
                k
                for k, v in namespace.items()
                if not k.startswith('__') and not k.endswith('__')
                    and not isinstance(v, (FunctionType, classmethod, staticmethod))
        ]
        return type.__new__(mcls, name, bases, namespace, **kwargs)

class NewAdsItem(metaclass=StaticOrderHelper):
    AdId        = Field()
    DateR       = Field()
    AdURL       = Field()

然后使用以下命令将_field_order项目导入到您的 piplines.py 中:

Then import the _field_order item into your piplines.py with:

...
from adbot.items import NewAdsItem
...
class DbPipeline(object):
    ikeys = NewAdsItem._field_order
    ...
    def createDbTable(self):
        print("Creating new table: %s" % self.dbtable )
        print("Keys in creatDbTable: \t%s" % ",".join(self.ikeys) )
        ...

现在，我可以按照正确的外观顺序创建新的数据库表，而不必担心Python以奇怪的方式对字典进行排序的怪异方式.

I can now create new DB tables in the correct order of appearance, without worrying of Python's weird way of sorting dicts in unexpected ways.

这篇关于如何以正确的顺序导入Scrapy项目密钥?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！