是否有任何 Python 模块可以帮助从 Javascript 加载的 DOM 中抓取数据? [英] Is there any Python module that helps to crawl data from DOM loaded by Javascript?

问题描述

我想从使用 Ajax 调用加载 DOM 元素的页面中抓取数据.

I want to scrape data from a page which loads DOM elements using Ajax call.

我尝试过旧的基于 PyQt4 的抓取解决方案，它在完全加载后加载 DOM，但问题是我需要做一个 POST 请求，它只能用于 GET.

I have tried with the old solution line PyQt4-based scraping, which loads the DOM after it's fully loaded, but the problem is that I need to do a POST request and it's only available for GET.

新的 Python 模块 ghost.py 存在超时问题:当它获取大型 DOM 树时它引发超时异常.

The new Python module ghost.py has time out issues: when it fetches a large DOM tree it raises a time out exception.

如果有人知道任何可以帮助我执行 POST 请求并在 DOM 完全加载后获取数据的特定方式或工具，那将对我有很大帮助.

If anyone knows any specific way or tools that can help me to do a POST request and grab the data after fully loaded DOM, that will help me a lot.

推荐答案

您可以使用 Selenium 来自动化浏览器和访问多姆.Selenium 有 python 驱动程序，因此你可以用 python 编写代码来导航到页面.单击按钮并等待 ajax 调用完成，然后再开始报废.

You can use Selenium to automate browser and access dom. Selenium has python driver hence you can write code in python to navigate to the page. click buttons and wait for ajax call to complete before you start scrapping.

这篇关于是否有任何 Python 模块可以帮助从 Javascript 加载的 DOM 中抓取数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！