单击 Scrapy 中的按钮 [英] Click a Button in Scrapy

问题描述

我正在使用 Scrapy 抓取网页.我需要的一些信息只有在你点击某个按钮时才会弹出(当然点击后的HTML代码中也会出现).

I'm using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

我发现 Scrapy 可以处理如下所示的表单(如登录)这里.但问题是没有表格可以填写，所以不是我需要的.

I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it's not exactly what I need.

如何简单地点击一个按钮，然后显示我需要的信息?

How can I simply click a button, which then shows the information I need?

我是否必须使用像 mechanize 或 lxml 这样的外部库?

Do I have to use an external library like mechanize or lxml?

推荐答案

Scrapy 无法解释 javascript.

Scrapy cannot interpret javascript.

如果您绝对必须与页面上的 javascript 交互，那么您希望使用 Selenium.

If you absolutely must interact with the javascript on the page, you want to be using Selenium.

如果使用 Scrapy，问题的解决方案取决于按钮的作用.

If using Scrapy, the solution to the problem depends on what the button is doing.

如果它只是显示以前隐藏的内容，你可以毫无问题地抓取数据，它不会出现在浏览器中并不重要，HTML 仍然存在.

If it's just showing content that was previously hidden, you can scrape the data without a problem, it doesn't matter that it wouldn't appear in the browser, the HTML is still there.

如果按下按钮时通过 AJAX 动态获取内容，最好的做法是使用 Firebug 等工具查看按下按钮时发出的 HTTP 请求.然后，您可以直接从该 URL 请求数据.

If it's fetching the content dynamically via AJAX when the button is pressed, the best thing to do is to view the HTTP request that goes out when you press the button using a tool like Firebug. You can then just request the data directly from that URL.

我是否必须使用像 mechanize 或 lxml 这样的外部库?

Do I have to use an external library like mechanize or lxml?

如果您想解释 javascript，是的，您需要使用不同的库，尽管这两个库都不符合要求.他们都对 javascript 一无所知.硒是要走的路.

If you want to interpret javascript, yes you need to use a different library, although neither of those two fit the bill. Neither of them know anything about javascript. Selenium is the way to go.

如果你能提供你正在抓取的页面的 URL，我可以看看.

If you can give the URL of the page you're working on scraping I can take a look.

这篇关于单击 Scrapy 中的按钮的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！