如何在 Ruby 中编写 Web 抓取工具? [英] How do I write a web scraper in Ruby?

问题描述

我想抓取一个没有 API 的流行网站(比如 Quora)并获取一些特定信息并将其转储到一个文件中 - 比如 csv、.txt 或 .html 格式很好:)

I would like to crawl a popular site (say Quora) that doesn't have an API and get some specific information and dump it into a file - say either a csv, .txt, or .html formatted nicely :)

例如仅返回 Quora 用户的所有简历"列表，这些用户在其公开信息中列出了用户体验设计师"职业.

E.g. return only a list of all the 'Bios' of the Users of Quora that have, listed in their publicly available information, the occupation 'UX designer'.

我将如何在 Ruby 中做到这一点?

How would I do that in Ruby ?

我对 Ruby &导轨工作.我刚刚完成了一个 Rails 应用程序——主要是我自己写的.但无论如何，我都不是大师.

I have a moderate enough level of understanding of how Ruby & Rails work. I just completed a Rails app - mainly all written by myself. But I am no guru by any stretch of the imagination.

我了解正则表达式等

推荐答案

最好的办法是使用 机械化.它可以跟随链接、提交表单、任何你需要的东西，网络客户端.顺便说一句，不要使用正则表达式来解析 HTML.使用 HTML 解析器.

Your best bet would be to use Mechanize.It can follow links, submit forms, anything you will need, web client-wise. By the way, don't use regexes to parse HTML. Use an HTML parser.

这篇关于如何在 Ruby 中编写 Web 抓取工具?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！