用JavaScript回发刮HTML [英] Scraping HTML with JavaScript postbacks

问题描述

我试图抓取一些HTML（得到作者的许可）。我正在使用PHP库这里建议，并且它运行良好，直到我遇到一个如下所示的链接：

 < a href =javascript：__ doPostBack（'dgItem $ _ctl2 $ _ctl0'，''）> 
  

我相信这是一些asp.net的东西。当我点击它时，它不会更改网址，它只是将一些新内容加载到页面中，我也希望这样做。

我怎样才能解决这个问题？

我想我需要模拟点击，但在处理原始HTML时我不能这样做，我需要某种浏览器/ JS解释器，没有？

是否有更适合此任务的库？我不局限于PHP，但它是首选。

解决方案

我最终将Python与Selenium Firefox web驱动程序结合使用。由于我使用的是真正的浏览器，因此我可以做一切FF。

I'm trying to scrape some HTML (with permission from the author). I was using the PHP library suggested here, and it was working well until I encountered a link that looks like this:

<a href="javascript:__doPostBack('dgItem$_ctl2$_ctl0','')">

Which I believe is some asp.net thing. When I click it, it doesn't change the URL, it just loads some new content into the page, which I'd also like to scrape.

How can I get around this?

I suppose I would need to simulate the click, but I can't do that when processing raw HTML, I'd need some kind of browser/JS interpreter, no?

Is there a better suited library for this task? I'm not limited to PHP, but it's preferred.

解决方案

I ended up using Python with Selenium Firefox web driver. Since I'm using a real browser, I can do everything FF can.

这篇关于用JavaScript回发刮HTML的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！