等待网页加载到Scrapy中 [英] Wait until the webpage loads in Scrapy

问题描述

我正在使用scrapy脚本使用"yield"加载URL.

I am using scrapy script to load URL using "yield".

MyUrl = "www.example.com"
request = Request(MyUrl, callback=self.mydetail)
yield request
def mydetail(self, response):
    item['Description'] = response.xpath(".//table[@class='list']//text()").extract()
    return item

该网址似乎至少需要5秒钟才能加载.因此，我希望Scrapy等待一段时间以将整个文本加载到item ['Description']中. 我在settings.py中尝试了"DOWNLOAD_DELAY"，但没有用.

The URL seems to take minimum 5 seconds to load. So I want Scrapy to wait for some time to load the entire text in item['Description']. I tried "DOWNLOAD_DELAY" in settings.py but no use.

推荐答案

简要介绍一下Firebug或其他工具来捕获对javascript代码所做的Ajax请求的响应.您可以做出一系列响应，以捕获在页面上传后出现的那些ajax请求.有几个相关的问题:重新整理最终页面， 解析动态内容.

Make a brief view on firebug or another tool to capture responses for Ajax requests, which were made by javascript code. You are able to make a chain of responses to catch those ajax requests which appear after uploading of the page.There are several related questions: parse ajax content, retreive final page, parse dynamic content.

这篇关于等待网页加载到Scrapy中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！