Scrapy 在提取时解释 html 实体 [英] Scrapy interpreting html entities on extract

问题描述

在抓取过程中，我通常以这种方式捕获链接:

During a crawling, I captured links usually that way:

response.xpath("//a[contains(@class, something)/@href").extract()

但由于某种原因，该特定页面无法正常工作.我在数组中收到的是这样的:

But for some reason in that specific page was not working. What I was receiving in the array was something like this:

['details?lm==true=1=A43', (...)]

正确的输出应该是:

['details?lm=&printerView=true&accessType=1&id=A43', (...)]

推荐答案

一段时间后，我发现 firefox 上的同一个页面渲染得很奇怪...我的问题一直在发生，因为被抓取的页面是 content-type作为文本/xml"而不是 html.

After sometime, I discovered that the same page on firefox was rendering weird... My problem has been happening because the page being crawled was with the content-type as "text/xml" and not html.

为了修复我的代码，我做了其他选择器:

To fix my code I did other selector:

sel = scrapy.Selector(text=response.body)
sel.xpath("//a[contains(@class, something)/@href").extract()

现在我得到了正确的结果！

And now I have the correct result!

['details?lm=&printerView=true&accessType=1&id=A43', (...)]

这篇关于Scrapy 在提取时解释 html 实体的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！