在单击事件"与调用&QUOT的问题;使用Python中美丽的汤在HTML页面上 [英] Issues with invoking "on click event" on the html page using beautiful soup in Python
问题描述
我想网页上刮的所有项目present的名字,但在默认情况下只有18页和放大器上可见;我的code仅那些刮。您可以通过点击全部显示按钮来查看所有项目,但该按钮是在Javascript。
I am trying to scrape names of all the items present on the webpage but by default only 18 are visible on the page & my code is scraping only those. You can view all items by clicking on "Show all" button but that button is in Javascript.
经过一番研究,我发现PyQt的模块可以用来解决涉及JavaScript的按钮和放这个问题;我用它,但我仍然无法调用上点击事件。下面是被称为code:
After some research, I found that PyQt module can be used to solve this issue involving javascript buttons & I used it but I am still not able to invoke the "on click" event. Below is the referred code:
import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
from PyQt4.QtGui import *
from PyQt4.QtCore import *
from PyQt4.QtWebKit import *
class Render(QWebPage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebPage.__init__(self)
self.loadFinished.connect(self._loadFinished)
self.mainFrame().load(QUrl(url))
self.app.exec_()
def _loadFinished(self, result):
self.frame = self.mainFrame()
self.app.quit()
url = 'http://www.att.com/shop/wireless/devices/smartphones.html'
r = Render(url)
jsClick = var evObj = document.createEvent('MouseEvents');
evObj.initEvent('click', true, true );
this.dispatchEvent(evObj);
allSelector = "a#deviceShowAllLink" # This is the css selector you actually need
allButton = r.frame.documentElement().findFirst(allSelector)
allButton.evaluateJavaScript(jsClick)
page = allButton
soup = BeautifulSoup(page)
soup.prettify()
with open('Smartphones_26decv1.0.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(["Date","Day of Week","Device Name","Price"])
items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=True)
prices = soup.findAll('div', {"class": "listGrid-price"})
for item, price in zip(items, prices):
textcontent = u' '.join(price.stripped_strings)
if textcontent:
spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%A") ,unicode(item.string).encode('utf8').strip(),textcontent])
我现在面临这个错误如下:
Error which I am facing in this is as follows:
"Invalid Syntax" Error for evObj
有人可以帮我在调用此的onclick事件,以便我能刮的所有数据items.Pardon我为我的无知,我是新来的节目。
Can someone please help me in invoking this "onclick" event so that I am able to scrape data for all items.Pardon me for my ignorance as I am new to programming.
推荐答案
要单击您必须调用按钮 evaluateJavascript
在对象:
To click the button you must call evaluateJavascript
over the object:
jsClick = """var evObj = document.createEvent('MouseEvents');
evObj.initEvent('click', true, true );
this.dispatchEvent(evObj);
"""
allSelector = "a#deviceShowAllLink" # This is the css selector you actually need
allButton = r.frame.documentElement().findFirst(allSelector)
allButton.evaluateJavaScript(jsClick)
这篇关于在单击事件"与调用&QUOT的问题;使用Python中美丽的汤在HTML页面上的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!