用python抓取javascript生成的html [英] scrape html generated by javascript with python

问题描述

我需要用 python 抓取一个站点.我使用 urlib 模块获取源 html 代码，但我还需要抓取一些由 javascript 函数(包含在 html 源代码中)生成的 html 代码.这个功能在网站中"的作用是，当您按下按钮时，它会输出一些 html 代码.如何使用 python 代码按下"这个按钮?刮刮乐可以帮助我吗?我用 firebug 捕获了 POST 请求，但是当我尝试在 url 上传递它时，我收到 403 错误.有什么建议吗?

I need to scrape a site with python. I obtain the source html code with the urlib module, but I need to scrape also some html code that is generated by a javascript function (which is included in the html source). What this functions does "in" the site is that when you press a button it outputs some html code. How can I "press" this button with python code? Can scrapy help me? I captured the POST request with firebug but when I try to pass it on the url I get a 403 error. Any suggestions?

推荐答案

在 Python 中，我认为 Selenium 1.0 是要走的路.这是一个库，可让您使用您选择的语言控制真实的网络浏览器.

In Python, I think Selenium 1.0 is the way to go. It’s a library that allows you to control a real web browser from your language of choice.

您需要在运行脚本的机器上安装有问题的网络浏览器，但它似乎是以编程方式查询使用大量 JavaScript 的网站的最可靠方法.

You need to have the web browser in question installed on the machine your script runs on, but it looks like the most reliable way to programmatically interrogate websites that use a lot of JavaScript.

这篇关于用python抓取javascript生成的html的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！