使用Selenium GeckoDriver和Firefox时是否可以减少内存RAM消耗 [英] Is it possible to reduce memory RAM consumption when using Selenium GeckoDriver and Firefox

问题描述

我将Selenium和Firefox Webdriver与python配合使用，以从网站中抓取数据.

I use Selenium and Firefox webdriver with python to scrape data from a website.

但是在代码中，我需要访问该网站超过1万次，并且这样做要消耗大量RAM.

But in the code, I need to access this website more than 10k times and it consumes a lot of RAM to do that.

通常，当脚本访问该站点2500次时，它已经消耗了4gb或更多的RAM，并且停止工作.

Usually, when the script access this site 2500 times, it already consumes 4gb or more of RAM and it stops to work.

是否可以在不关闭浏览器会话的情况下减少内存RAM的消耗?

Is it possible to reduce memory RAM consumption without close browser session?

我问这是因为，当我启动脚本时，需要手动登录该站点(双因子自动加法，下面未显示代码)，并且如果我关闭浏览器会话，则需要登录该站点再次.

I ask that because when I start the script, I need to log manually on the site(two-factor autentication, the code is not shown below) and if I close the browser session, I will need to log in the site again.

for itemLista in lista:
    driver.get("https://mytest.site.com/query/option?opt="+str(itemLista))

    isActivated = driver.find_element_by_xpath('//div/table//tr[2]//td[1]')
    activationDate = driver.find_element_by_xpath('//div/table//tr[2]//td[2]')

    print(str(isActivated.text))
    print(str(activationDate.text))

    indice+=1
    print("numero: "+str(indice))

    file2.write(itemLista+" "+str(isActivated.text)+" "+str(activationDate.text)+"\n")

#close file
file2.close()

推荐答案

我发现了如何避免内存泄漏.

I discover how to avoid the memory leak.

我只是用

time.sleep(2)

之后

file2.write(itemLista+" "+str(isActivated.text)+" "+str(activationDate.text)+"\n")

现在firefox可以在不消耗大量RAM的情况下运行了

Now firefox is working without consumes lots of RAM

那太完美了.

我不知道为什么停止它会消耗这么多内存，但是我认为它正在增加内存消耗，因为它没有时间来完成每个driver.get请求.

I don't know exactly why it stopped consumes so much memory, but I think it was growing memory consume because it didn't have time to finish each driver.get request.

这篇关于使用Selenium GeckoDriver和Firefox时是否可以减少内存RAM消耗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！