使用请求HTTP库了解python中的内存消耗增加 [英] Understanding memory consumption increase in python using Requests HTTP library

问题描述

我正在使用python v2.7和请求HTTP库.我正在使用Windows 7 OS.

I am using python v2.7 and Requests HTTP library. I am using windows 7 OS.

我无法理解为什么以下代码在执行时会消耗越来越多的内存?我已经观察到使用任务管理器的内存消耗.如果有人可以指出可能的原因以及如何避免它，那将是很好的.如果有人可以在他/她的系统上对其进行测试并确认不只是我自己或我的系统面临此问题，那也很好. :)尽管内存消耗量会少量增加，但我认为在没有剩余可用内存的情况下，我的应用肯定会崩溃.

I have failed to understand why the following code consumes more and more memory upon execution? I have observed the memory consumption using task manager. It would be great if some one can point out what could be the possible reason and how to avoid it? Also it would be great if some one can test this on his/her system and confirm that it is not only me or my system who is facing this problem. :) Although the memory consumption increases with a small amount, I think at some point my app will surely crash when there is no more free memory left.

代码:

import requests
def getName():
    url = 'https://stackoverflow.com/users/2382792/pss'
    r = requests.get(url)
    print r

while True:
    getName()

下面将详细说明此简单问题的原因.另外，我在 urllib2模块中也遇到了类似的问题.我想到了使用请求HTTP库.但是关于内存消耗的相同问题仍然存在.

The reason for this simple question is provided in detail below. Also I have a similar problem with urllib2 module too. So I thought of using Requests HTTP library. But the same problem regarding the memory consumption persists.

现实世界场景:我的现实世界应用程序是使用python和wxPython构建的. wxPython代码不会导致此问题.连接到URL以便从服务器获取某些值的python代码引起了问题.为了保持值的更新，每隔2秒就会获取一次值.然后将这些获取的值显示在GUI上.上面显示的代码是我原始代码的示例.在以上代码示例中，此外，只有一种连接到URL的实例. In my real world app I have 9-10 threads those are doing this fetching stuff by connecting to different URLs! Which means in this case the memory consumption increases more rapidly as compared to the above sample code. 我希望现在更加清楚这个问题的严重性.另外，为澄清起见，我不会删除任何网站.

Real world scenario: My real world app is built with python and wxPython. The wxPython code doesn't causes this problem. The python code which is connecting to the URLs to fetch some values from the server is causing the problems. To keep the values updated the values are fetched after every 2 seconds. These fetched values are then displayed on the GUI. The code shown above is the a sample of my original code. More over in above code sample there is only one instance of connecting to an URL. In my real world app I have 9-10 threads those are doing this fetching stuff by connecting to different URLs! Which means in this case the memory consumption increases more rapidly as compared to the above sample code. I hope now it is more clear how severe this problem is. Also, just for the clarification I am not scrapping any website.

比较:为什么以下代码与上面的代码没有相同的问题:

Comparison: Why the following code doesn't have the same problem as the code above:

import random
def getValue():
    value = random.randrange(0,11)
    print value
while True:
    getValue()

更新:我测试了半个小时的代码段.我最后得出的结论是，内存消耗在开始时就增加了，后来又变成了既减少又增加的状态.最终，内存消耗将保持在一定范围内. 这是从Windows 8 OS上的 processExplorer 创建的图形.中间一个用于内存(右键单击图像，然后单击查看以清楚地看到它):

Update: I tested the code snippet for half an hour. I have finally concluded that the memory consumption increases in the beginning and later it gets in a state where it decreases as well as increases. Finally the memory consumption will remain in a range. Here is a graph created from processExplorer on Windows 8 OS. The middle one is for the memory(Right click the image and click view to see it clearly):

谢谢您的时间！

推荐答案

我测试了半个小时的代码段.我最后得出的结论是，内存消耗在开始时就增加了，后来又变成了既减少又增加的状态.最终，内存消耗将保持在一定范围内.我认为这是阅读有用的注释后系统应该工作的方式. :)

I tested the code snippet for half an hour. I have finally concluded that the memory consumption increases in the beginning and later it gets in a state where it decreases as well as increases. Finally the memory consumption will remain in a range. I believe this is how the system is supposed to work after reading the useful comments. :)

这篇关于使用请求HTTP库了解python中的内存消耗增加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！