Scrapy 和 Selenium 提交动态呈现的表单 [英] Scrapy and Selenium submit form that renders dynamically

问题描述

我使用 selenium 首先加载通过 ajax 生成的表单.

现在我在将 selenium 响应传递给scrapy FormReuqest 方法时遇到了麻烦

发送表单数据值.

表单在用户提交之前有jquery验证，使用scrapy提交会不会更难?

感谢任何帮助.

谢谢

解决方案

这里既不需要 Scrapy，也不需要 selenium.只需发出一个底层 POST 请求并解析 json 响应.使用 requests 的示例:

导入json进口请求URL = 'http://calculator.shipito.com/en/rates'数据 = {位置":10"，国家":GB"，城市":伦敦"，"邮政编码": "123456",包":[{dimensions_units":in"，weight_units":lbs"，dimension":{width":9"，height":20.4"，length":17"},重量":9"，价值":170"}]}response = requests.post(URL, data=json.dumps(data))打印 response.json()

打印:

<预><代码>[{u'infoUrl': u'http://www.shipito.com/postage#tnt', u'isPromoted': False, u'popularity': 1, u'name': u'TNT Economy', u'DeliveryTime': 6, u'insurable': True, u'usesDimWeight': True, u'shippingRate': 68.38, u'separateShippingRate': None, u'bonusShippingRate': 68.38, u'deliveryTimeInfo': u'4-6工作日', u'mps': False, u'insuranceRate': 0, u'logoImg': u'tnt.png'},...]

I'm using selenium to first load the form which being generated via ajax.

Now I'm having troubles passing the selenium response to scrapy FormReuqest method

to send the form data values.

The form has jquery validation before user can submit it, does it make it harder to submit using scrapy?

any help is appreciated.

thanks

解决方案

You don't need neither Scrapy, nor selenium here. Just make an underlying POST request and parse the json response. Example using requests:

import json
import requests


URL = 'http://calculator.shipito.com/en/rates'
data = {"location": "10", "country": "GB", "city": "London",
        "postalcode": "123456", 
        "packages": [{"dimensions_units": "in", "weight_units": "lbs", "dimension": {"width": "9", "height": "20.4", "length": "17"},
        "weight": "9", "value": "170"}]}

response = requests.post(URL, data=json.dumps(data))
print response.json()

Prints:

[
    {u'infoUrl': u'http://www.shipito.com/postage#tnt', u'isPromoted': False, u'popularity': 1, u'name': u'TNT Economy', u'deliveryTime': 6, u'insurable': True, u'usesDimWeight': True, u'shippingRate': 68.38, u'separateShippingRate': None, u'bonusShippingRate': 68.38, u'deliveryTimeInfo': u'4-6 business days', u'mps': False, u'insuranceRate': 0, u'logoImg': u'tnt.png'}, 
    ...
]

这篇关于Scrapy 和 Selenium 提交动态呈现的表单的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！