如何发送在Scrapy中启用的JavaScript和Cookies? [英] How to send JavaScript and Cookies Enabled in Scrapy?

问题描述

我正在使用Scrapy抓取一个网站，该网站需要启用烹饪和Java脚本.我认为我不必实际处理javascript.我只需要假装好像启用了javascript.

I am scraping a website using Scrapy which require cooking and java-script to be enabled. I don't think I will have to actually process javascript. All I need is to pretend as if javascript is enabled.

这是我尝试过的: 1)通过以下设置启用Cookie

Here is what I have tried: 1) Enable Cookies through following in settings

COOKIES_ENABLED = True
COOKIES_DEBUG = True

2)使用下载中间件获取Cookie

2) Using download middleware for cookies

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
    'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':700
}

3)发送"X-JAVASCRIPT-ENABLED":"True"

3) Sending 'X-JAVASCRIPT-ENABLED': 'True'

DEFAULT_REQUEST_HEADERS={
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
    'X-JAVASCRIPT-ENABLED': 'True'
}

但是他们都没有和我一起工作.您能提出任何想法或给我一些指导吗?

but none of them is working with me. Can you please suggest any idea or give me some direction ?

预先感谢您的答复.

推荐答案

您应该尝试 Splash scrapyjs 的JS引擎.这是一个如何在您的Spider项目中进行设置的示例:

You should try Splash JS engine with scrapyjs. Here is a example of how to set it up in your spider project:

SPLASH_URL = 'http://192.168.59.103:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapyjs.SplashMiddleware': 725,
}

与Scrapy背后的公司相同的

抓取中心具有

Scraping hub which is the same company behind Scrapy, has special instances to run your spiders with splash enabled.

然后像这样在蜘蛛中生成SplashRequest而不是Request

Then yield SplashRequest instead of Request in your spider like this:

import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):
    start_urls = ["http://example.com", "http://example.com/foo"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse,
                endpoint='render.html',
                args={'wait': 0.5},
            )

    def parse(self, response):
        # response.body is a result of render.html call; it
        # contains HTML processed by a browser.
        # …

这篇关于如何发送在Scrapy中启用的JavaScript和Cookies?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！