应用引擎(python)如何跨请求管理内存(超出了软私有内存限制) [英] How does app engine (python) manage memory across requests (Exceeded soft private memory limit)

问题描述

我在App Engine中的各种请求处理程序中偶尔遇到Exceeded soft private memory limit错误.我知道此错误意味着实例使用的RAM已超出分配的数量，以及这如何导致实例关闭.

I'm experiencing occasional Exceeded soft private memory limit error in a wide variety of request handlers in app engine. I understand that this error means that the RAM used by the instance has exceeded the amount allocated, and how that causes the instance to shut down.

我想了解导致错误的可能原因，并想从头开始了解应用引擎python实例应如何管理内存.我的基本假设是:

I'd like to understand what might be the possible causes of the error, and to start, I'd like to understand how app engine python instances are expected to manage memory. My rudimentary assumptions were:

1. 一个F2实例以256 MB开头
2. 启动时，它会加载我的应用程序代码-比方说30 MB
3. 处理请求时，它有226 MB可用空间
   • 只要该请求不超过226 MB(+错误余量)，该请求就会完成且没有错误
   • 如果它确实超过了226 MB +余量，则实例完成请求，记录超出软私有内存限制"错误，然后终止-现在返回步骤1

1. An F2 instance starts with 256 MB
2. When it starts up, it loads my application code - lets say 30 MB
3. When it handles a request it has 226 MB available
   • so long as that request does not exceed 226 MB (+ margin of error) the request completes w/o error
   • if it does exceed 226 MB + margin, the instance completes the request, logs the 'Exceeded soft private memory limit' error, then terminates - now go back to step 1

这是我假定的工作方式，但是鉴于我偶尔偶尔会在相当宽的请求处理程序集范围内看到此错误，我现在不太确定.我的问题是:

That's how I presumed it would work, but given that I'm occasionally seeing this error across a fairly wide set of request handlers, I'm now not so sure. My questions are:

a)步骤4是否发生?

a) Does step #4 happen?

b)什么会导致它不发生?还是不完全发生?例如请求之间的内存泄漏怎么可能?

b) What could cause it not to happen? or not to fully happen? e.g. how could memory leak between requests?

c)模块级变量中的存储是否会导致内存使用率泄漏? (我不是故意以这种方式使用模块级变量)

c) Could storage in module level variables causes memory usage to leak? (I'm not knowingly using module level variables in that way)

d)我可以使用哪些工具/技术来获取更多数据?例如.测量进入请求处理程序时的内存使用情况?

d) What tools / techniques can I use to get more data? E.g. measure memory usage at entry to request handler?

在可能的答案/评论中，请链接到gae文档.

In answers/comments, where possible, please link to the gae documentation.

[edit]额外信息:我的应用配置为threadsafe: false.如果这与答案有关，请说明答案.我计划尽快更改为threadsafe: true.

[edit] Extra info: my app is congifured as threadsafe: false. If this has a bearing on the answer, please state what it is. I plan to change to threadsafe: true soon.

[edit] 说明:该问题与用于内存管理的gae的预期行为有关.因此，尽管诸如呼叫gc.collect()"之类的建议很可能是相关问题的部分解决方案，但它们并未完全回答该问题.直到我了解gae的预期行为为止，使用gc.collect()对我来说就像是伏都教编程.

[edit] Clarification: This question is about the expected behavior of gae for memory management. So while suggestions like 'call gc.collect()' might well be partial solutions to related problems, they don't fully answer this question. Up until the point that I understand how gae is expected to behave, using gc.collect() would feel like voodoo programming to me.

最后:如果我把这一切都弄糟了，那我就提前道歉-我真的找不到很多有用的信息，所以我主要是在猜测..

Finally: If I've got this all backwards then I apologize in advance - I really cant find much useful info on this, so I'm mostly guessing..

推荐答案

与其他任何标准的Python解释器相比，App Engine的Python解释器在内存管理方面没有任何特殊之处.因此，尤其是，每个请求"没有什么特别的事情发生，例如您的假设步骤4.相反，一旦任何对象的引用计数减少到零，Python解释器就会回收该内存(模块gc仅存在于此)处理垃圾循环-当一堆对象从未获得引用时，由于没有相互访问的外部引用，即使彼此引用，计数也降为零).

App Engine's Python interpreter does nothing special, in terms of memory management, compared to any other standard Python interpreter. So, in particular, there is nothing special that happens "per request", such as your hypothetical step 4. Rather, as soon as any object's reference count decreases to zero, the Python interpreter reclaims that memory (module gc is only there to deal with garbage cycles -- when a bunch of objects never get their reference counts down to zero because they refer to each other even though there is no accessible external reference to them).

因此，如果您使用任何全局变量，则内存很容易在请求之间泄漏"(实际上，尽管从技术上讲，这不是泄漏)–所说的变量将在处理程序类及其实例(例如)中保留下来. c6>方法-即您的观点(c)，尽管您说自己没有这样做.

So, memory could easily "leak" (in practice, though technically it's not a leak) "between requests" if you use any global variable -- said variables will survive the instance of the handler class and its (e.g) get method -- i.e, your point (c), though you say you are not doing that.

一旦您声明模块为threadsafe，一个实例可能碰巧同时服务多个请求(直到您在模块.yaml配置文件的automatic_scaling部分中设置为max_concurrent_requests的实例为止)；默认值为8).因此，您实例的RAM将是每个请求所需的RAM的倍数.

Once you declare your module to be threadsafe, an instance may happen to serve multiple requests concurrently (up to what you've set as max_concurrent_requests in the automatic_scaling section of your module's .yaml configuration file; the default value is 8). So, your instance's RAM will need be a multiple of what each request needs.

关于(d)，要获取更多数据"(我想您实际上是想获得更多的RAM)，唯一可以做的就是为需要内存的模块配置更大的instance_class.

As for (d), to "get more data" (I imagine you actually mean, get more RAM), the only thing you can do is configure a larger instance_class for your memory-hungry module.

要使用更少的RAM ，有很多技术-与App Engine无关，与Python无关，尤其是与非常特定的代码及其代码无关.非常具体的需求.

To use less RAM, there are many techniques -- which have nothing to do with App Engine, everything to do with Python, and in particular, everything to do with your very specific code and its very specific needs.

我能想到的一个与GAE相关的问题是，据报道ndb的缓存泄漏了-请参阅

The one GAE-specific issue I can think of is that ndb's caching has been reported to leak -- see https://code.google.com/p/googleappengine/issues/detail?id=9610 ; that thread also suggests workarounds, such as turning off ndb caching or moving to old db (which does no caching and has no leak). If you're using ndb and have not turned off its caching, that might be the root cause of "memory leak" problems you're observing.

这篇关于应用引擎(python)如何跨请求管理内存(超出了软私有内存限制)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！