Scrapy:URL错误,程序添加了不必要的字符(URL代码) [英] Scrapy: URL error, Program adds unnecessary characters(URL-codes)
问题描述
我使用Scrapyto搜寻了一个德语论坛: http://www.musikerboard.de/forum
im using Scrapyto crawl a german forum: http://www.musikerboard.de/forum
它遵循所有子论坛并从线程中提取信息.
It follows all subforums and extracts Information from threads.
问题:在抓取过程中,它在多个线程链接上给我一个错误:
The problem: During crawling it gives me an error on ultiple threadlinks:
2015-09-26 14:01:59 [scrapy] DEBUG: Ignoring response <404 http://www.musiker-board.de/threads/spotify-premium-paket.621224/%0A%09%09>: HTTP status code is not handled or not allowed
除了这部分/%0A%09%09
它会显示404错误.
我不知道为什么程序会不断将代码添加到URL的末尾
I dont know why the program keeps adding the code to the end of the URL
这里是我的代码:
def urlfunc(value):
value = value.replace("%0A", "")
value = value.replace("%09", "")
return value
class spidermider(CrawlSpider):
name = 'memberspider'
allowed_domains = ["musiker-board.de"]
start_urls = ['http://www.musiker-board.de/forum/'
# 'http://www.musiker-board.de/'
] # urls from which the spider will start crawling
rules = (
Rule(LinkExtractor(allow=(r'forum/\w*',))),
Rule(LinkExtractor(allow=(r'threads/\w+',),deny=(r'threads/\w+/[\W\d]+'),process_value=urlfunc), callback='parse_thread' ),
)
有人解释为什么它不断发生吗(及其解决方案)
Does someone have a explanation why it keeps happening?(And a solution to it)
更新的代码
推荐答案
如果进行一些手动调试和研究,您会发现URL末尾的值是元字符. %0A
是换行符,%09
是水平制表符: http://www.w3schools .com/tags/ref_urlencode.asp
If you do some manual debugging and research you will find that the values at the end of the URL are meta-characters. %0A
is a line feed, %09
is a horizontal tab: http://www.w3schools.com/tags/ref_urlencode.asp
然后,如果您使用手动调试语句丰富了urlfunc
函数(并将日志级别提高到INFO
以更好地查看结果),那么您将看到URL不会仅以字符串作为字符结尾称为网站时会被转换.
Then if you enrich your urlfunc
function with manual debug statements (and increase the log-level to INFO
to see the results better) then you will see that the URLs do not end with these characters as a string just are converted when calling it as a website.
def urlfunc(value):
print 'orgiginal: ', value
value = value.replace('%0A', '').replace('%09', '')
print 'replaced: ', value
return value
此结果显示在以下输出中:
This resulst in the following output:
orgiginal: http://www.musiker-board.de/posts/7609325/
replaced: http://www.musiker-board.de/posts/7609325/
orgiginal: http://www.musiker-board.de/members/martin-hofmann.17/
replaced: http://www.musiker-board.de/members/martin-hofmann.17/
第一个结果和第二个结果之间的线在输出中,因为它们具有元字符.
The lines between the first result and the second one are there in the output because they have the meta-characters.
所以解决方案是strip
值:
def urlfunc(value):
return value.strip()
在这种情况下,您不会收到任何调试消息,告诉您找不到该站点.
In this case you do not get any debug messages which tell you that the site was not found.
这篇关于Scrapy:URL错误,程序添加了不必要的字符(URL代码)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!