强制 Python Scrapy 不编码 URL [英] Force Python Scrapy not to encode URL
问题描述
有一些带有 []
的 URL,比如
There are some URLs with []
in it like
http://www.website.com/CN.html?value_ids[]=33&value_ids[]=5007
但是当我尝试使用 Scrapy 抓取此 URL 时,它会向此 URL 发出请求
But when I try scraping this URL with Scrapy, it makes Request to this URL
http://www.website.com/CN.html?value_ids%5B%5D=33&value_ids%5B%5D=5007
如何强制scrapy不对我的URL进行urlenccode?p>
How can I force scrapy to not to urlenccode my URLs?
推荐答案
在创建 Request
对象时,scrapy 应用了一些 url 编码方法.要恢复这些,您可以使用自定义中间件并根据需要更改 url.
When creating a Request
object scrapy applies some url encoding methods. To revert these you can utilize a custom middleware and change the url to your needs.
您可以像这样使用 Downloader Middleware
:
class MyCustomDownloaderMiddleware(object):
def process_request(self, request, spider):
request._url = request.url.replace("%5B", "[", 2)
request._url = request.url.replace("%5D", "]", 2)
不要忘记像这样激活"settings.py
中的中间件:
Don't forget to "activate" the middleware in settings.py
like so:
DOWNLOADER_MIDDLEWARES = {
'so.middlewares.MyCustomDownloaderMiddleware': 900,
}
我的项目名为 so
,文件夹中有一个文件 middlewares.py
.您需要根据您的环境进行调整.
My project is named so
and in the folder there is a file middlewares.py
. You need to adjust those to your environment.
这篇关于强制 Python Scrapy 不编码 URL的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!