如何从 url 中删除查询? [英] How do I remove a query from a url?
本文介绍了如何从 url 中删除查询?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在使用scrapy 来抓取一个站点,该站点似乎在每个URL 末尾的查询字符串中附加了随机值.这将爬行变成了一种无限循环.
I am using scrapy to crawl a site which seems to be appending random values to the query string at the end of each URL. This is turning the crawl into a sort of an infinite loop.
如何让scrapy忽略URL的查询字符串部分?
How do i make scrapy to neglect the query string part of the URL's?
推荐答案
示例代码:
from urlparse import urlparse
o = urlparse('http://url.something.com/bla.html?querystring=stuff')
url_without_query_string = o.scheme + "://" + o.netloc + o.path
示例输出:
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from urlparse import urlparse
>>> o = urlparse('http://url.something.com/bla.html?querystring=stuff')
>>> url_without_query_string = o.scheme + "://" + o.netloc + o.path
>>> print url_without_query_string
http://url.something.com/bla.html
>>>
这篇关于如何从 url 中删除查询?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文