在Google身份验证后抓取页面 [英] Fetch pages with scrapy behind Google Authentication

问题描述

我正在尝试登录使用Google凭据的网站.这在我的蜘蛛网中失败了:

I'm trying to log into a website that uses Google credentials. This fails in my scrapy spider:

def parse(self, response):
    return scrapy.FormRequest.from_response(
        response,
        formdata={'email': self.var.user, 'password': self.var.password},
        callback=self.after_login)

有什么提示吗?

推荐答案

经过进一步检查，我设法解决了一个似乎很简单的问题:

After further inspection I managed to solve this, seems to be, a simple issue:

1. 字段依次为电子邮件和 Passwd .
2. 将登录分为两个请求，第一个用于电子邮件，第二个用于密码.

起作用的代码如下:

def parse(self, response):
    """
    Insert the email. Next, go to the password page.
    """
    return scrapy.FormRequest.from_response(
        response,
        formdata={'Email': self.var.user},
        callback=self.log_password)


def log_password(self, response):
    """
    Enter the password to complete the log in.
    """
    return scrapy.FormRequest.from_response(
        response,
        formdata={'Passwd': self.var.password},
        callback=self.after_login)

这篇关于在Google身份验证后抓取页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！