使用 Ruby on Rails 从 URL 创建动态站点地图 [英] Create dynamic sitemap from URL with Ruby on Rails

问题描述

我目前正在开发一个应用程序，我从多个不同的站点抓取信息.要在站点上获取所需主题的深层链接，我依赖于提供的站点地图(例如论坛").在我扩展的过程中，我遇到了一些自己不提供站点地图的站点，所以我想知道是否有任何方法可以在 Rails 中从顶级域生成它?

I am currently working on an application where I scrape information from a number of different sites. To get the deeplink for the desired topic on a site I rely on the sitemap that is provided (e.g. "Forum"). As I am expanding I came across some sites that don't provide a sitemap themselves, so I was wondering if there was any way to generate it within Rails from the top level domain?

我正在使用 Nokogiri 和 Mechanize 来检索数据，因此如果有任何功能可以帮助解决该任务，那么集成会更容易.

I am using Nokogiri and Mechanize to retrieve data, so if there is any functionality that could help to tackle that task it would be easier to integrate.

推荐答案

这可以通过 Spidr gem 像这样:

This can be done with the Spidr gem like so:

url_map = Hash.new { |hash,key| hash[key] = [] }

Spidr.site('http://intranet.com/') do |spider|
  spider.every_link do |origin,dest|
    url_map[dest] << origin
  end
end

这篇关于使用 Ruby on Rails 从 URL 创建动态站点地图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！