使 Scrapy 从 Javascript 函数发送 POST 数据 [英] Make Scrapy send POST data from Javascript function

问题描述

我正在玩 Scrapy 并玩这个

并在scrapy中复制它:

cookies = {'wants_mature_content':'1','出生时间':'189302401','lastagecheckage': '1-January-1976',}url = 'https://store.steampowered.com/app/9200/RAGE/'请求(网址，cookie)

lastagecheckage 本身应该就足够了，但我还没有测试过.

I'm playing with Scrapy and playing with this tutorial. Things look good but I noticed Steam changed their age check so there is no longer a form in DOM. So the suggested solution will not work:

form = response.css('#agegate_box form')

action = form.xpath('@action').extract_first()

name = form.xpath('input/@name').extract_first()

value = form.xpath('input/@value').extract_first()

formdata = {

        name: value,

        'ageDay': '1',

        'ageMonth': '1',

        'ageYear': '1955'

}

yield FormRequest(

        url=action,

        method='POST',

        formdata=formdata,

        callback=self.parse_product

    )

Checking an example game that forces age check; I noticed the View Page button is no longer a form:

<a class="btnv6_blue_hoverfade btn_medium" href="#" onclick="ViewProductPage()"><span>View Page</span></a>

And the function being called will eventually call this one:

function CheckAgeGateSubmit( callbackFunc )
    {
                    if ( $J('#ageYear').val() == 2019 )
        {
            ShowAlertDialog( '', 'Please enter a valid date' );
            return false;
        }

        $J.post(
            'https://store.steampowered.com/agecheckset/' + "app" + '/9200/',
            {
                sessionid: g_sessionID,
                ageDay: $J('#ageDay').val(),
                ageMonth: $J('#ageMonth').val(),
                ageYear: $J('#ageYear').val()
            }
        ).done( function( response ) {
            switch ( response.success )
            {
                case 1:
                    callbackFunc();
                    break;
                case 24:
                    top.location.reload();
                    break;
                case 15:
                case 2:
                    ShowAlertDialog( 'Error', 'There was a problem verifying your age.  Please try again later.' );
                    break;
            }
        } );

}

So basically this is making a POST with some data...what would be the best way to do this in Scrapy, since this is not a form any longer? I'm just thinking on ignoring the code where the form is obtained and simply send the request with the FormRequest object...but is this the way to go? An alternative could also be setting cookies for age and pass it on every single request so possibly the age check is ignored altogether?

Thanks!

解决方案

You should probably just set an appropriate cookie and you'll be let right through!

If you take a look at what your browser has when entering the page:

and replicate that in scrapy:

cookies = {
    'wants_mature_content':'1',
    'birthtime':'189302401',
    'lastagecheckage': '1-January-1976',
}
url = 'https://store.steampowered.com/app/9200/RAGE/'
Request(url, cookies)

lastagecheckage should probably be enough on it's own but I haven't tested it.

这篇关于使 Scrapy 从 Javascript 函数发送 POST 数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！