如何忽略网络爬虫中的文件类型? [英] How do I ignore file types in a web crawler?

问题描述

我正在编写一个网络爬虫，想忽略链接到二进制文件的 URL:

I'm writing a web crawler and want to ignore URLs which link to binary files:

$exclude = %w(flv swf png jpg gif asx zip rar tar 7z gz jar js css dtd xsd ico raw mp3 mp4 wav wmv ape aac ac3 wma aiff mpg mpeg avi mov ogg mkv mka asx asf mp2 m1v m3u f4v pdf doc xls ppt pps bin exe rss xml)

如何根据这些结尾之一检查 URI?

How can I check the URI against one of these endings?

@url = URI.parse(url)

如果不包含上述任何后缀，则应设置.

should be set if it doesn't contain any of the suffixes above.

推荐答案

use URI#path:

use URI#path:

unless URI.parse(url).path =~ /\.(\w+)$/ && $exclude.include?($1)
  puts "downloading #{url}..."
end

这篇关于如何忽略网络爬虫中的文件类型?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！