在Python中使用Selenium刮除无限滚动网站 [英] Scraping infinite scrolling website with Selenium in Python

问题描述

我想剪贴这个带有滚动功能的网站的内容: http://stocktwits.com/symbol/AAPL?q=AAPL

I would like to scrap the content of this website that has scrolling: http://stocktwits.com/symbol/AAPL?q=AAPL

我在Stactoverflow中找到了类似问题的答案: 以无限滚动方式刮擦网站

I found this answer for a similar question in Stactoverflow: scrape websites with infinite scrolling

这是从那里复制的代码:

and here is the code copied from there:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import sys

import unittest, time, re

class Sel(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://twitter.com"
        self.verificationErrors = []
        self.accept_next_alert = True
    def test_sel(self):
        driver = self.driver
        delay = 3
        driver.get(self.base_url + "/search?q=stckoverflow&src=typd")
        driver.find_element_by_link_text("All").click()
        for i in range(1,100):
            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(4)
        html_source = driver.page_source
        data = html_source.encode('utf-8')


if __name__ == "__main__":
    unittest.main()

现在我要代替Stocktwits网站而不是Twitter(链接在上面).

Now instead of twitter I want to scrap Stocktwits website (the link is above).

我将上面的代码修改为此:

I modified the above code to this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import sys

import unittest, time, re

class Sel(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "http://stocktwits.com/symbol/AAPL?q=AAPL"
        self.verificationErrors = []
        self.accept_next_alert = True
    def test_sel(self):
        driver = self.driver
        delay = 3
        driver.get(self.base_url)
        driver.find_element_by_link_text("All").click()
        for i in range(1,100):
            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(4)
        html_source = driver.page_source
        data = html_source.encode('utf-8')


if __name__ == "__main__":
    unittest.main()

但是当我运行代码时，出现此错误:

But when I run the code I get this error:

NoSuchElementException: Message: Unable to locate element: {"method":"link text","selector":"All"}

我非常感谢您找出问题所在.

I appreciate any help to find out what is wrong.

推荐答案

问题似乎出在这一行:

driver.find_element_by_link_text("All").click()

您期望包含链接文本为"All"的元素，但不存在.

You're expecting an element with link text "All" but none exists.

这篇关于在Python中使用Selenium刮除无限滚动网站的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！