屏幕上刮擦鼠标悬停时使用Scrapy和硒的Datepicker [英] Screen scraping a Datepicker with Scrapy and Selenium on mouse hover

问题描述

所以我需要删除一个页面，如这个，我正在使用Scrapy + Seleninum与日期选择器日历进行交互。

我意识到，如果一定日期是可用的工具提示显示的价格，如果它不可用，如果你悬停在它没有任何事情。

我的代码是什么获得动态出现的价格当您在可用的日期悬停时，还如何知道它是否可用或不仅与悬停？

解决方案

不是那么简单的如何处理这个问题，因为页面的动态性质 - 你必须使用等待在这里和那里捕捉点击或悬停上出现的动态组件的HTML是棘手的。

这是comp lete工作代码将导航到页面，单击签入输入，等待日历加载并报告日历中的每一天的可用性（它使用 ui- datepicker-unselectable 类来确定）。然后，它使用 move_to_element（）浏览器操作来悬停每个单元格，等待工具提示并获取价格：

<$来自selenium import webdriver的$ p code从selenium.webdriver导入
从selenium.webdriver.common.by导入的ActionChains
来自selenium.webdriver.support.ui的
从selenium.webdriver.support导入WebDriverWait
将import expected_conditions作为EC


driver = webdriver.Firefox（）
driver.get（https：// www。 airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29）

＃等待支票输入加载
wait = WebDriverWait（driver，10）
elem = wait.until（EC.visibility_of_element_located（（By.CSS_SELECTOR，div.book-it-panel input [name = checkin]）））
elem .click（）

＃等待datepicker加载
wait.until（
EC.visibility_of_element_located（（By.CSS_SELECTOR，'.ui-datepicker：not（.loading） '））
）

days = driver.find_elements_by_css_ selector（。ui-datepicker table.ui-datepicker-calendar tr td）
表示单元格天数：
day = cell.text.strip（）
如果不是日期：
continue

如果cell.get_attribute（class）中的ui-datepicker-unselectable：
status =不可用
else：
status = 可用

price =n / a
如果status ==可用：
＃悬停单元格并等待工具提示
ActionChains（驱动程序）.move_to_element（cell）.perform（）
price = wait.until（EC.visibility_of_element_located（（By.CSS_SELECTOR，'.datepicker-tooltip'）））。text

print（日，状态，价格）

打印：

  1不可用n / a 
 2不可用n / a 
 3不可用n / a 
 4不可用n / a 
 5不可用n / a 
 6不可用n / a 
 7不可用n / a 
 8不可用n / a 
 9不可用n / a 
 10不可用n / a 
 11不可用n / a 
 12 Unava ilable n / a 
 13可用€40 
 14不可用n / a 
 15不可用n / a 
 16不可用n / a 
 17不可用n / a 
 18不可用n / a 
 19可用€36 
 20可用€49 
 21不可用n / a 
 22可用€49 
 23不可用n / a 
 24不可用n / a 
 25可用€40 
 26可用€39 
 27可用€35 
 28可用€37 
 29可用€37 
 30可用€37 
  

So I need to scrap a page like this for example and I am using Scrapy + Seleninum to interact with a date-picker calendar.

I realized that if a certain date is available a price shows on the tooltip, and if its not available if you hover on it nothing happens.

Whats the code for me to get the price that appears dynamically when you hover on an available day and also how do I know if its available or not just with the hover?

解决方案

It is not that straightforward how to approach the problem because of the dynamic nature of the page - you have to use waits here and there and it's tricky to catch the HTML of the dynamic components appearing on click or hover.

Here is the complete working code that would navigate to the page, click the "Check In" input, wait for the calendar to load and report the availability for each of the days in the calendar (it uses the presence of ui-datepicker-unselectable class to determine that). Then, it hovers each cell using the move_to_element() browser action, waits for the tooltip and gets the price:

from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get("https://www.airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29")

# wait for the check in input to load
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.book-it-panel input[name=checkin]")))
elem.click()

# wait for datepicker to load
wait.until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, '.ui-datepicker:not(.loading)'))
)

days = driver.find_elements_by_css_selector(".ui-datepicker table.ui-datepicker-calendar tr td")
for cell in days:
    day = cell.text.strip()
    if not day:
        continue

    if "ui-datepicker-unselectable" in cell.get_attribute("class"):
        status = "Unavailable"
    else:
        status = "Available"

    price = "n/a"
    if status == "Available":
        # hover the cell and wait for the tooltip
        ActionChains(driver).move_to_element(cell).perform()
        price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.datepicker-tooltip'))).text

    print(day, status, price)

Prints:

1 Unavailable n/a
2 Unavailable n/a
3 Unavailable n/a
4 Unavailable n/a
5 Unavailable n/a
6 Unavailable n/a
7 Unavailable n/a
8 Unavailable n/a
9 Unavailable n/a
10 Unavailable n/a
11 Unavailable n/a
12 Unavailable n/a
13 Available €40
14 Unavailable n/a
15 Unavailable n/a
16 Unavailable n/a
17 Unavailable n/a
18 Unavailable n/a
19 Available €36
20 Available €49
21 Unavailable n/a
22 Available €49
23 Unavailable n/a
24 Unavailable n/a
25 Available €40
26 Available €39
27 Available €35
28 Available €37
29 Available €37
30 Available €37

这篇关于屏幕上刮擦鼠标悬停时使用Scrapy和硒的Datepicker的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！