Scrapy:测试内联请求的有效方法 [英] Scrapy: effective way to test inline requests

问题描述

我使用scrapy-inline-requests 库编写了一个spider.所以我的蜘蛛中的解析方法看起来像这样:

I wrote a spider using scrapy-inline-requests library. So the parse method in my spider looks something like this:

@inline_requests
def parse(self, response1):
    item = MyItem()
    loader = ItemLoader(item=item, response=response1)

    #extracting some data from the response1

    try:
        response 2 = yield Request(some_url)
        #extracting some other data from response2
    except Exception:
            self.logger.warning("Failed request to: %s", some_url)

    yield loader.load_item()

我想有效地测试这个方法.我可以轻松编写一个测试，在其中创建一个假模拟 response1 并将其传递给函数.但是，我不知道如何模拟 response2 并使用来自虚假响应的数据获取完整项目.您有什么建议吗?

I want to effectively test this method. I can easily write a test, in which a create a fake mock response1 and pass it to the function. However, I've got no idea how to mock response2 and get the complete item with the data from both the fake responses. Do you have any suggestions?

推荐答案

可能有点晚，但请查看 scrapy-inline-requests 的 github 存储库中的测试:https://github.com/rmax/scrapy-inline-requests/blob/master/tests/test_inline_requests.py.

Might be a bit late but check out the tests in scrapy-inline-requests' github repo: https://github.com/rmax/scrapy-inline-requests/blob/master/tests/test_inline_requests.py.

基本上，按照您的回调将产生对这些响应的请求的顺序，向回调提供一个 Response 对象列表.

Basically, feed the callback a list of Response objects in the order that your callback will yield Requests for these responses.

这篇关于Scrapy:测试内联请求的有效方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！