如何从加载主页后加载特定数据的页面中抓取数据? [英] How do I scrape data from a page that loads specific data after the main page load?

问题描述

我一直在使用 Ruby 和 Nokogiri 从类似于 hollister 网站的 URL 中提取数据:http://www.hollisterco.com/webapp/wcs/stores/servlet/TrackDetail?storeId=10251&;catalogId=10201&langId=-1&URL=TrackDetailView&orderNumber=1316358

I have been using Ruby and Nokogiri to pull data from a URL similar to this one from the hollister website: http://www.hollisterco.com/webapp/wcs/stores/servlet/TrackDetail?storeId=10251&catalogId=10201&langId=-1&URL=TrackDetailView&orderNumber=1316358

我的脚本现在看起来像这样:

My script looks like this right now:

require 'rubygems'
require 'nokogiri'
require 'open-uri'

page = Nokogiri::HTML(open("http://www.hollisterco.com/webapp/wcs/stores/servlet/TrackDetail?storeId=10251&catalogId=10201&langId=-1&URL=TrackDetailView&orderNumber=1316358")) 

puts page.css("h3[data-property=GLB_ORDERNUMBERSYMBOL]")[0].text

我的问题是 Hollister 页面有某种异步加载数据，因此当我的脚本使用页面元素的订单特定数据检查页面区域时，它还不存在.IE，带有 data-property=GBL_ORDERNUMBERSYMBOL 的 <h3> 还不存在，但是在浏览器中，如果让它再加载十秒钟，DOM 和HTML 更改以反映特定的订单详细信息.

My problem is that the Hollister page has some sort of asynchronous loading of data, such that when my script checks the area of the page with order specific data for a page element, it doesn't exist yet. I.E., the <h3> with data-property=GBL_ORDERNUMBERSYMBOL doesn't yet exist, but in the browser if you let it load for another ten seconds, the DOM and HTML change to reflect the specific order details.

捕获事后加载的数据的最佳方法是什么?我曾尝试使用 watir-webdriver，但不确定我需要做什么才能使该驱动程序正常工作.

What is the best way to capture this data that loads after the fact? I have tried using the watir-webdriver, but not sure what I would need to do to make that one work either.

推荐答案

我不知道如何使用 Open-URI 来实现，但是如果您想使用 Watir-Webdriver，则以下方法有效.

I am not sure how to do it with Open-URI, but if you want to use Watir-Webdriver, the following works.

require 'watir-webdriver'
b = Watir::Browser.new
b.goto('http://www.hollisterco.com/webapp/wcs/stores/servlet/TrackDetail?storeId=10251&catalogId=10201&langId=-1&URL=TrackDetailView&orderNumber=1316358')
puts b.h3(:class, 'order-num').when_present.text

请注意，在 h3 标记上执行了 when_present().这意味着脚本将在尝试获取其文本之前等待 h3 出现.如果您知道有些部分需要时间来加载，那么添加显式等待通常可以解决问题.

Note that a when_present() is performed on the h3 tag. What this means is that the script will wait for the h3 to appear before trying to get its text. If you know there are parts that take time to load, adding an explicit wait usually solves the problem.

这篇关于如何从加载主页后加载特定数据的页面中抓取数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！