我想使用GAE刮取网站并将结果发布到Google实体中 [英] I want to scrape a site using GAE and post the results into a Google Entity

问题描述

我想抓取此网址： https：// www.xstreetsl.com/modules.php?searchSubmitImage_x=0&searchSubmitImage_y=0&SearchLocale=0&name=Marketplace&SearchKeyword=business&searchSubmitImage.x=0&searchSubmitImage.y=0&SearchLocale=0&SearchPriceMin=& SearchPriceMax =& SearchRatingMin =& SearchRatingMax =& sort =& dir = asc

进入每个链接并提取出各个部分信息例如然后将结果发布到Google应用引擎上的实体中。

我想知道最好的方法吗？

Chris

解决方案

为了使用纯Python库对HTML进行规范化处理， href =http://code.google.com/p/html5lib/ =nofollow noreferrer> html5lib 比BeautifulSoup。

然而，你只是想提取简单的结构化信息，这实际上并不需要对HTML进行标准化。我在Google App Engine上有一些使用我自己的xpath库与原始HTML一起使用。
或者您可以使用正则表达式进行一次关闭作业。

I want to scrape this URL : https://www.xstreetsl.com/modules.php?searchSubmitImage_x=0&searchSubmitImage_y=0&SearchLocale=0&name=Marketplace&SearchKeyword=business&searchSubmitImage.x=0&searchSubmitImage.y=0&SearchLocale=0&SearchPriceMin=&SearchPriceMax=&SearchRatingMin=&SearchRatingMax=&sort=&dir=asc

Go into each of the links and extract out various pieces of information e.g. permissions, prims etc then post the results into a Entity on google app engine.

I want to know the best way to go about it?

Chris

解决方案

For normalizing HTML using a pure Python library I have had better experiences with html5lib than BeautifulSoup.

However you just want to extract simply structured information, which doesn't actually require normalizing the HTML. I have a few scraping apps on Google App Engine which use my own xpath library that works with raw HTML. Or you can use regular expressions for one off jobs.

这篇关于我想使用GAE刮取网站并将结果发布到Google实体中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！