如何在 Scrapy 中更新 DjangoItem [英] How to update DjangoItem in Scrapy

问题描述

我一直在使用 Scrapy，但遇到了一些问题.

I've been working with Scrapy but run into a bit of a problem.

DjangoItem 有一个 save 方法来使用 Django ORM 持久化项目.这很棒，但如果我多次运行刮刀，即使我可能只想更新以前的值，也会在数据库中创建新项目.

DjangoItem has a save method to persist items using the Django ORM. This is great, except that if I run a scraper multiple times, new items will be created in the database even though I may just want to update a previous value.

查看文档和源代码后，我看不到任何更新现有项目的方法.

After looking at the documentation and source code, I don't see any means to update existing items.

我知道我可以调用 ORM 来查看某个项目是否存在并更新它，但这意味着调用数据库以获取每个单个对象，然后再次调用以保存该项目.

I know that I could call out to the ORM to see if an item exists and update it, but it would mean calling out to the database for every single object and then again to save the item.

如果项目已经存在，我该如何更新?

How can I update items if they already exist?

推荐答案

不幸的是，我发现实现此目的的最佳方法是完全按照说明执行操作:使用 django_model 检查该项目是否存在于数据库中.objects.get，然后更新它.

Unfortunately, the best way that I found to accomplish this is to do exactly what was stated: Check if the item exists in the database using django_model.objects.get, then update it if it does.

在我的设置文件中，我添加了新管道:

In my settings file, I added the new pipeline:

ITEM_PIPELINES = {
    # ...
    # Last pipeline, because further changes won't be saved.
    'apps.scrapy.pipelines.ItemPersistencePipeline': 999
}

我创建了一些辅助方法来处理创建项目模型的工作，并在必要时创建一个新模型:

I created some helper methods to handle the work of creating the item model, and creating a new one if necessary:

def item_to_model(item):
    model_class = getattr(item, 'django_model')
    if not model_class:
        raise TypeError("Item is not a `DjangoItem` or is misconfigured")

    return item.instance


def get_or_create(model):
    model_class = type(model)
    created = False

    # Normally, we would use `get_or_create`. However, `get_or_create` would
    # match all properties of an object (i.e. create a new object
    # anytime it changed) rather than update an existing object.
    #
    # Instead, we do the two steps separately
    try:
        # We have no unique identifier at the moment; use the name for now.
        obj = model_class.objects.get(name=model.name)
    except model_class.DoesNotExist:
        created = True
        obj = model  # DjangoItem created a model for us.

    return (obj, created)


def update_model(destination, source, commit=True):
    pk = destination.pk

    source_dict = model_to_dict(source)
    for (key, value) in source_dict.items():
        setattr(destination, key, value)

    setattr(destination, 'pk', pk)

    if commit:
        destination.save()

    return destination

然后，最终的管道相当简单:

Then, the final pipeline is fairly straightforward:

class ItemPersistencePipeline(object):
    def process_item(self, item, spider):
        try:
             item_model = item_to_model(item)
        except TypeError:
            return item

        model, created = get_or_create(item_model)

        update_model(model, item_model)

        return item

这篇关于如何在 Scrapy 中更新 DjangoItem的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！