使用 Requests HTTP 库了解 Python 中的内存消耗增加 [英] Understanding memory consumption increase in python using Requests HTTP library

问题描述

我正在使用 python v2.7 和 请求 HTTP 库.我使用的是 Windows 7 操作系统.

I am using python v2.7 and Requests HTTP library. I am using windows 7 OS.

我不明白为什么下面的代码在执行时会消耗越来越多的内存?我使用任务管理器观察了内存消耗.如果有人能指出可能的原因以及如何避免它，那就太好了?如果有人可以在他/她的系统上对此进行测试并确认不仅仅是我或我的系统面临这个问题，那也会很棒.:) 虽然内存消耗会以少量增加，但我认为在某些时候，当没有更多可用内存时，我的应用程序肯定会崩溃.

I have failed to understand why the following code consumes more and more memory upon execution? I have observed the memory consumption using task manager. It would be great if some one can point out what could be the possible reason and how to avoid it? Also it would be great if some one can test this on his/her system and confirm that it is not only me or my system who is facing this problem. :) Although the memory consumption increases with a small amount, I think at some point my app will surely crash when there is no more free memory left.

代码:

import requests
def getName():
    url = 'https://stackoverflow.com/users/2382792/pss'
    r = requests.get(url)
    print r

while True:
    getName()

下面详细提供了这个简单问题的原因.我也有与 urllib2 模块类似的问题. 所以我想到了使用 Requests HTTP 库.但是关于内存消耗的同样问题仍然存在.

The reason for this simple question is provided in detail below. Also I have a similar problem with urllib2 module too. So I thought of using Requests HTTP library. But the same problem regarding the memory consumption persists.

真实世界场景:我的真实世界应用程序是用 python 和 wxPython 构建的.wxPython 代码不会导致此问题.连接到 URL 以从服务器获取某些值的 python 代码导致了问题.为了保持值更新，每 2 秒获取一次值.这些获取的值随后会显示在 GUI 上.上面显示的代码是我的原始代码示例.在上面的代码示例中，只有一个连接到 URL 的实例.在我现实世界的应用程序中，我有 9-10 个线程，它们通过连接到不同的 URL 来执行此操作！这意味着在这种情况下，与上述示例代码相比，内存消耗增加得更快. 我希望现在更清楚这个问题有多严重.另外，为了澄清起见，我不会废弃任何网站.

Real world scenario: My real world app is built with python and wxPython. The wxPython code doesn't causes this problem. The python code which is connecting to the URLs to fetch some values from the server is causing the problems. To keep the values updated the values are fetched after every 2 seconds. These fetched values are then displayed on the GUI. The code shown above is the a sample of my original code. More over in above code sample there is only one instance of connecting to an URL. In my real world app I have 9-10 threads those are doing this fetching stuff by connecting to different URLs! Which means in this case the memory consumption increases more rapidly as compared to the above sample code. I hope now it is more clear how severe this problem is. Also, just for the clarification I am not scrapping any website.

对比:为什么下面的代码和上面的代码没有同样的问题:

Comparison: Why the following code doesn't have the same problem as the code above:

import random
def getValue():
    value = random.randrange(0,11)
    print value
while True:
    getValue()

更新:我测试了半个小时的代码片段.我终于得出结论，内存消耗在开始时会增加，后来它会处于既减少又增加的状态.最后内存消耗会保持在一个范围内.这是在 Windows 8 操作系统上从 processExplorer 创建的图表.中间一个是内存(右击图片，点view可以看清楚):

Update: I tested the code snippet for half an hour. I have finally concluded that the memory consumption increases in the beginning and later it gets in a state where it decreases as well as increases. Finally the memory consumption will remain in a range. Here is a graph created from processExplorer on Windows 8 OS. The middle one is for the memory(Right click the image and click view to see it clearly):

感谢您的宝贵时间！

推荐答案

我测试了半个小时的代码片段.我终于得出结论，内存消耗在开始时会增加，后来它会处于既减少又增加的状态.最后内存消耗会保持在一个范围内.我相信这是系统在阅读了有用的评论后应该如何工作的.:)

I tested the code snippet for half an hour. I have finally concluded that the memory consumption increases in the beginning and later it gets in a state where it decreases as well as increases. Finally the memory consumption will remain in a range. I believe this is how the system is supposed to work after reading the useful comments. :)

这篇关于使用 Requests HTTP 库了解 Python 中的内存消耗增加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！