在新标签页中打开网页 Selenium + Python [英] Open web in new tab Selenium + Python

问题描述

所以我试图在我的 WebDriver 中的新选项卡上打开网站.我想这样做，因为使用 PhantomJS 为每个网站打开一个新的 WebDriver 大约需要 3.5 秒，我想要更快的速度...

So I am trying to open websites on new tabs inside my WebDriver. I want to do this, because opening a new WebDriver for each website takes about 3.5secs using PhantomJS, I want more speed...

我使用的是多进程python脚本，我想从每个页面获取一些元素，所以工作流程是这样的:

I'm using a multiprocess python script, and I want to get some elements from each page, so the workflow is like this:

Open Browser

Loop throught my array
For element in array -> Open website in new tab -> do my business -> close it

但我找不到任何方法来实现这一目标.

But I can't find any way to achieve this.

这是我正在使用的代码.网站之间需要永远，我需要它很快......允许使用其他工具，但我不知道有太多工具可以用于抓取使用 JavaScript 加载的网站内容(在加载时触发某些事件时创建的 div 等)为什么我需要 Selenium... BeautifulSoup 不能用于我的某些页面.

Here's the code I'm using. It takes forever between websites, I need it to be fast... Other tools are allowed, but I don't know too many tools for scrapping website content that loads with JavaScript (divs created when some event is triggered on load etc) That's why I need Selenium... BeautifulSoup can't be used for some of my pages.

#!/usr/bin/env python
import multiprocessing, time, pika, json, traceback, logging, sys, os, itertools, urllib, urllib2, cStringIO, mysql.connector, shutil, hashlib, socket, urllib2, re
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from PIL import Image
from os import listdir
from os.path import isfile, join
from bs4 import BeautifulSoup
from pprint import pprint

def getPhantomData(parameters):
    try:
        # We create WebDriver
        browser = webdriver.Firefox()
        # Navigate to URL
        browser.get(parameters['target_url'])
        # Find all links by Selector
        links = browser.find_elements_by_css_selector(parameters['selector'])

        result = []
        for link in links:
            # Extract link attribute and append to our list
            result.append(link.get_attribute(parameters['attribute']))
        browser.close()
        browser.quit()
        return json.dumps({'data': result})
    except Exception, err:
        browser.close()
        browser.quit()
        print err

def callback(ch, method, properties, body):
    parameters = json.loads(body)
    message = getPhantomData(parameters)

    if message['data']:
        ch.basic_ack(delivery_tag=method.delivery_tag)
    else:
        ch.basic_reject(delivery_tag=method.delivery_tag, requeue=True)

def consume():
    credentials = pika.PlainCredentials('invitado', 'invitado')
    rabbit = pika.ConnectionParameters('localhost',5672,'/',credentials)
    connection = pika.BlockingConnection(rabbit)
    channel = connection.channel()

    # Conectamos al canal
    channel.queue_declare(queue='com.stuff.images', durable=True)
    channel.basic_consume(callback,queue='com.stuff.images')

    print ' [*] Waiting for messages. To exit press CTRL^C'
    try:
        channel.start_consuming()
    except KeyboardInterrupt:
        pass

workers = 5
pool = multiprocessing.Pool(processes=workers)
for i in xrange(0, workers):
    pool.apply_async(consume)

try:
    while True:
        continue
except KeyboardInterrupt:
    print ' [*] Exiting...'
    pool.terminate()
    pool.join()

推荐答案

编者注:此答案不再适用于新的 Selenium 版本.请参阅此评论.

Editor's note: This answer no longer works for new Selenium versions. Refer to this comment.

您可以通过组合键COMMAND + T 或COMMAND + W (OSX).在其他操作系统上，您可以使用 CONTROL + T/CONTROL + W.

You can achieve the opening/closing of a tab by the combination of keys COMMAND + T or COMMAND + W (OSX). On other OSs you can use CONTROL + T / CONTROL + W.

在 selenium 中，您可以模拟这种行为.您将需要创建一个网络驱动程序和与您需要的测试一样多的选项卡.

In selenium you can emulate such behavior. You will need to create one webdriver and as many tabs as the tests you need.

这是代码.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://www.google.com/")

#open tab
driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 't') 
# You can use (Keys.CONTROL + 't') on other OSs

# Load a page 
driver.get('http://stackoverflow.com/')
# Make the tests...

# close the tab
# (Keys.CONTROL + 'w') on other OSs.
driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 'w') 


driver.close()

这篇关于在新标签页中打开网页 Selenium + Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！