如何使用硒同时打开多个Firefox实例而不清除缓存和cookie [英] How to use selenium to open multiple instances Firefox simultaneously without clearing cache and cookie
问题描述
我有一个代码,可从Facebook UID抓取朋友列表。它确实有效,但要花费很长时间才能抓取整个列表。因此,我想通过使用多处理和Selenium Grid来加快速度。以下是我使用的方法:
I have a code which scrape friend list from Facebook UID. It worked but it takes a long time to scrape a whole list. So, I want to speed it up by using multiprocessing and Selenium Grid. The following is the approach I use:
- 使用帐户登录Facebook
- 使用5个实例打开Firefox相同的缓存和Cookie(因此我无需再次登录)
- 同时从5个不同的UID中抓取朋友列表。 1个实例/ 1个UID
这是我的代码,但是不起作用
This is my code but it doesn't work
import multiprocessing
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
from selenium import webdriver
def friend_uid_list(uid, driver):
driver.get('https://www.facebook.com/' + uid + '/friends')
//scrape friend list
target.close()
def g(arg):
return friend_uid_list(*arg)
if __name__ == '__main__':
driver = webdriver.Firefox()
driver.get("https://www.facebook.com/")
driver.find_element_by_css_selector("#email").send_keys("email@gmail.com")
driver.find_element_by_css_selector("#pass").send_keys("password")
driver.find_element_by_css_selector("#u_0_m").click()
pool = multiprocessing.Pool(5)
pool.map(g, [(100004159542140,driver),(100004159542140,driver),(100004159542140,driver)])
那么,您能告诉我如何使用Selenium Grid同时使用多个实例吗?我进行了很多搜索,但不知道如何将其实现到我的代码中。谢谢:)
So, can you show me how to use Selenium Grid to use multiple instances simultaneously ? I searched a lot but don't know how to implement it to my code. Thank you :)
推荐答案
这是不使用硒网格的另一种方法。
Here is another approach without using selenium grid.
此方法将打开5个Firefox实例,以及每个实例上的3个窗口。 cookie是从主实例复制过来的。
This approach opens 5 firefox instances, as well as 3 windows on each instance. The cookies are copied over from the main instance.
from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import multiprocessing
display = Display(visible=0, size=(800, 600))
display.start()
d = webdriver.Firefox()
def friend_uid_list(uid, driver):
values = []
for handle in driver.window_handles:
driver.switch_to_window(handle)
# driver.wait_for_element() etc etc
values.append(driver.find_element_by_id('#something'))
# scrape elements
return values
def g(arg):
return friend_uid_list(*arg)
启动实例并登录:
d = webdriver.Firefox()
d.get("https://www.facebook.com/")
d.find_element_by_css_selector("#email").send_keys("email@gmail.com")
d.find_element_by_css_selector("#pass").send_keys("password")
d.find_element_by_css_selector("#loginbutton").click()
启动多个实例:
drivers = [webdriver.Firefox(), webdriver.Firefox(), webdriver.Firefox(), webdriver.Firefox()]
复制 localStorage
:
localstorage_kv = d.execute_script("var obj={};for (var i=0,len=localStorage.length;i<len;++i){obj[localStorage.key(i)]=localStorage.getItem(localStorage.key(i));};return obj")
复制cookie和 localStorage
:
for e in drivers:
e.get("https://www.facebook.com/")
for x in d.get_cookies():
e.add_cookie(x)
for k, v in localstorage_kv.items():
e.execute_script('localStorage.setItem("{}", {})'.format(k,v))
e.refresh() # should be logged in now
添加初始驱动程序返回驱动程序数组:
Add the initial driver back into the drivers array:
drivers.append(d)
然后遍历uid:
uids = [100004159542140, 100004159542140, 100004159542140, 100004159542140, 100004159542140, 100004159542140]
pool = multiprocessing.Pool(5)
while uids:
for driver in drivers:
if len(driver.window_handles) == 1:
driver.execute_script('window.open("https://www.facebook.com/' + uids.pop() + '/friends")')
driver.execute_script('window.open("https://www.facebook.com/' + uids.pop() + '/friends")')
else:
for handle in driver.window_handles:
handle.get("https://www.facebook.com/" + uids.pop() + "/friends")
return_values = pool.map(g, drivers)
import pdb;pdb.set_trace()
如果您真的想跨节点共享cookie在硒g上摆脱外观:
If you really want to share the cookies across nodes on a selenium grid look:
- 在硒网格上创建并上传文件
- 如何使用python selenium webdriver保存和加载Cookie
- http://selenium-python.readthedocs.io/getting-started.html#selenium -remote-webdriver
- Create and upload a file on selenium grid
- How to save and load cookies using python selenium webdriver
- http://selenium-python.readthedocs.io/getting-started.html#selenium-remote-webdriver
这大致上是指腌制 localStorage
和 cookies
并将其传输到每个节点,然后从那里将cookie读入每个节点上的每个实例。
Which roughly means pickle the localStorage
and cookies
and transfer that to each node, from there then read the cookie into each instance on each node.
这篇关于如何使用硒同时打开多个Firefox实例而不清除缓存和cookie的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!