Google ndb 库中的内存泄漏 [英] Memory leak in Google ndb library
问题描述
我认为 ndb 库中存在内存泄漏，但我找不到位置.
I think there is a memory leak in the ndb library but I can not find where.
有没有办法避免下面描述的问题?
您是否有更准确的测试想法来找出问题所在?
Is there a way to avoid the problem described below?
Do you have a more accurate idea of testing to figure out where the problem is?
这就是我重现问题的方式:
That's how I reproduced the problem :
我用 2 个文件创建了一个极简的 Google App Engine 应用程序.app.yaml:
I created a minimalist Google App Engine application with 2 files.
app.yaml:
application: myapplicationid
version: demo
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /.*
script: main.APP
libraries:
- name: webapp2
version: latest
main.py:
# -*- coding: utf-8 -*-
"""Memory leak demo."""
from google.appengine.ext import ndb
import webapp2
class DummyModel(ndb.Model):
content = ndb.TextProperty()
class CreatePage(webapp2.RequestHandler):
def get(self):
value = str(102**100000)
entities = (DummyModel(content=value) for _ in xrange(100))
ndb.put_multi(entities)
class MainPage(webapp2.RequestHandler):
def get(self):
"""Use of `query().iter()` was suggested here:
https://code.google.com/p/googleappengine/issues/detail?id=9610
Same result can be reproduced without decorator and a "classic"
`query().fetch()`.
"""
for _ in range(10):
for entity in DummyModel.query().iter():
pass # Do whatever you want
self.response.headers['Content-Type'] = 'text/plain'
self.response.write('Hello, World!')
APP = webapp2.WSGIApplication([
('/', MainPage),
('/create', CreatePage),
])
我上传了一次名为 /create 的应用程序.
之后，每次调用 / 都会增加实例使用的内存.直到它由于错误而停止 Exceeded soft private memory limit of 128 MB with 143 MB after service 5 requests total.
I uploaded the application, called /create once.
After that, each call to / increases the memory used by the instance. Until it stops due to the error Exceeded soft private memory limit of 128 MB with 143 MB after servicing 5 requests total.
内存使用图示例(您可以看到内存增长和崩溃):
Exemple of memory usage graph (you can see the memory growth and crashes) :
注意:这个问题可以用webapp2以外的其他框架重现，比如web.py
Note: The problem can be reproduced with another framework than webapp2, like web.py
推荐答案
经过更多的调查，并在 google 工程师的帮助下，我找到了两个解释我的内存消耗的原因.
After more investigations, and with the help of a google engineer, I've found two explanation to my memory consumption.
上下文和线程
ndb.Context 是一个线程本地"对象，只有在线程中有新请求时才会清除.所以线程在请求之间保持它.一个 GAE 实例中可能存在许多线程，并且在第二次使用线程并清除上下文之前可能需要数百个请求.
这不是内存泄漏，但内存中的上下文大小可能会超过小型 GAE 实例中的可用内存.
ndb.Context is a "thread local" object and is only cleared when a new request come in the thread. So the thread hold on it between requests. Many threads may exist in a GAE instance and it may take hundreds of requests before a thread is used a second time and it's context cleared.
This is not a memory leak, but contexts size in memory may exceed the available memory in a small GAE instance.
解决方法:
您无法配置 GAE 实例中使用的线程数.因此，最好使每个上下文尽可能小.避免上下文缓存，并在每次请求后清除它.
Workaround:
You can not configure the number of threads used in a GAE instance. So it is best to keep each context smallest possible. Avoid in-context cache, and clear it after each request.
事件队列
似乎 NDB 并不能保证在请求后清空事件队列.同样，这不是内存泄漏.但是它将 Futures 留在您的线程上下文中，您又回到了第一个问题.
It seems that NDB does not guarantee that event queue is emptied after a request. Again this is not a memory leak. But it leave Futures in your thread context, and you're back to the first problem.
解决方法:
用 @ndb.toplevel 包装所有使用 NDB 的代码.
Workaround:
Wrap all your code that use NDB with @ndb.toplevel.
这篇关于Google ndb 库中的内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
