调用“点击事件"的问题在 html 页面上使用 Python 中的美丽汤 [英] Issues with invoking "on click event" on the html page using beautiful soup in Python

问题描述

我正在尝试抓取网页上所有项目的名称，但默认情况下，页面上只有 18 个可见 &我的代码只抓取那些.您可以通过单击全部显示"按钮查看所有项目，但该按钮在 Javascript 中.

I am trying to scrape names of all the items present on the webpage but by default only 18 are visible on the page & my code is scraping only those. You can view all items by clicking on "Show all" button but that button is in Javascript.

经过一番研究，我发现PyQt模块可以用来解决这个涉及javascript按钮的问题&我使用了它，但我仍然无法调用点击"事件.下面是引用的代码:

After some research, I found that PyQt module can be used to solve this issue involving javascript buttons & I used it but I am still not able to invoke the "on click" event. Below is the referred code:

import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  

class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()  

url = 'http://www.att.com/shop/wireless/devices/smartphones.html'  
r = Render(url)
jsClick = var evObj = document.createEvent('MouseEvents');
             evObj.initEvent('click', true, true );
             this.dispatchEvent(evObj);


allSelector = "a#deviceShowAllLink" # This is the css selector you actually need
allButton   = r.frame.documentElement().findFirst(allSelector)
allButton.evaluateJavaScript(jsClick)




page = allButton
soup = BeautifulSoup(page)
soup.prettify()
with open('Smartphones_26decv1.0.csv', 'wb') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=',')
    spamwriter.writerow(["Date","Day of Week","Device Name","Price"])
    items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=True)
    prices = soup.findAll('div', {"class": "listGrid-price"})
    for item, price in zip(items, prices):
        textcontent = u' '.join(price.stripped_strings)
        if textcontent:            
            spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%A") ,unicode(item.string).encode('utf8').strip(),textcontent])

我在这方面面临的错误如下:

Error which I am facing in this is as follows:

"Invalid Syntax" Error for evObj

有人可以帮助我调用这个onclick"事件，以便我能够抓取所有项目的数据.请原谅我的无知，因为我是编程新手.

Can someone please help me in invoking this "onclick" event so that I am able to scrape data for all items.Pardon me for my ignorance as I am new to programming.

推荐答案

要单击按钮，您必须在对象上调用 evaluateJavascript:

To click the button you must call evaluateJavascript over the object:

jsClick = """var evObj = document.createEvent('MouseEvents');
             evObj.initEvent('click', true, true );
             this.dispatchEvent(evObj);
             """

allSelector = "a#deviceShowAllLink" # This is the css selector you actually need
allButton   = r.frame.documentElement().findFirst(allSelector)
allButton.evaluateJavaScript(jsClick)

这篇关于调用“点击事件"的问题在 html 页面上使用 Python 中的美丽汤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！