有没有任何Java脚本Web爬虫框架 [英] is there any java script web crawler framework
本文介绍了有没有任何Java脚本Web爬虫框架的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
是否有任何JavaScript网络爬虫框架?
Is there any JavaScript web crawler framework?
推荐答案
有一个刚刚发布的新框架Node.js名为< a href =https://github.com/mikeal/spider>蜘蛛。它使用jQuery来抓取/索引网站的HTML页面。 API和配置非常好,特别是如果你已经知道jQuery。
There's a new framework that was just release for Node.js called spider. It uses jQuery under the hood to crawl/index a website's HTML pages. The API and configuration are really nice especially if you already know jQuery.
从测试套件中,这里有一个爬行纽约时报网站的例子:
From the test suite, here's an example of crawling the New York Times website:
var spider = require('../main');
spider()
.route('www.nytimes.com', '/pages/dining/index.html', function (window, $) {
$('a').spider();
})
.route('travel.nytimes.com', '*', function (window, $) {
$('a').spider();
if (this.fromCache) return;
var article = { title: $('nyt_headline').text(), articleBody: '', photos: [] }
article.body = ''
$('div.articleBody').each(function () {
article.body += this.outerHTML;
})
$('div#abColumn img').each(function () {
var p = $(this).attr('src');
if (p.indexOf('ADS') === -1) {
article.photos.push(p);
}
})
console.log(article);
})
.route('dinersjournal.blogs.nytimes.com', '*', function (window, $) {
var article = {title: $('h1.entry-title').text()}
console.log($('div.entry-content').html())
})
.get('http://www.nytimes.com/pages/dining/index.html')
.log('info')
;
这篇关于有没有任何Java脚本Web爬虫框架的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文