如何在scrapy蜘蛛的start_urls中发送post数据 [英] How to send post data in start_urls of the scrapy spider

问题描述

我想抓取一个仅支持发布数据的网站.我想发送查询参数在所有请求中发布数据.如何实现这一目标?

I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this?

推荐答案

POST 请求可以使用 scrapy 的 Request 或 FormRequest 类.

POST requests can be made using scrapy's Request or FormRequest classes.

另外，考虑使用 start_requests() 方法而不是 start_urls 属性.

Also, consider using start_requests() method instead of start_urls property.

示例:

from scrapy.http import FormRequest

class myspiderSpider(Spider):
    name = "myspider"
    allowed_domains = ["www.example.com"]

    def start_requests(self):
        return [ FormRequest("http://www.example.com/login",
                     formdata={'someparam': 'foo', 'otherparam': 'bar'},
                     callback=self.parse) ]

希望有所帮助.

这篇关于如何在scrapy蜘蛛的start_urls中发送post数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！