Google App Engine/数据存储区/Flask/Python应用程序中的内存泄漏 [英] Memory leak in Google App Engine / Datastore / Flask / Python app

问题描述

我建立了一个简单的新闻聚合器网站，其中所有App Engine实例的内存使用量一直在增长，直到达到极限并因此被关闭.

I have built a simple news aggregator site, in which the memory usage of all my App Engine instances keep growing until reaching the limit and therefore being shut down.

我已经开始消除应用程序中的所有内容，以提供最小的可复制版本.这就是我现在拥有的:

I have started to eliminate everything from my app to arrive at a minimal reproducible version. This is what I have now:

app = Flask(__name__)

datastore_client = datastore.Client()

@app.route('/')
def root():
    
    query = datastore_client.query(kind='source')
    query.order = ['list_sequence']
    sources = query.fetch() 
    
    for source in sources:
        pass
    

统计数据显示了一种典型的锯齿模式:在实例启动时，它达到190-210 Mb，然后在某些请求(不是全部请求)下，内存使用量增加了20-30 Mb. (顺便说一句，虽然我不确定这是否是相关信息，但大致相当于查询结果的估计大小.)这种情况一直持续到关闭时它超过512 Mb.它通常发生在对"/"的请求的第50至100左右.在此期间，没有其他要求.

Stats show a typical saw-tooth pattern: at instance startup, it goes to 190 - 210 Mb, then upon some requests, but NOT ALL requests, memory usage increases by 20 - 30 Mb. (This, by the way, roughly corresponds to the estimated size of the query results, although I cannot be sure this is relevant info.) This keeps happening until it exceeds 512 Mb, when it is shut down. It usually happens at around the 50th - 100th request to "/". No other requests are made to anything else in the meantime.

现在，如果我删除"for"循环，仅查询仍然存在，问题消失了，内存使用率保持在190 Mb不变，即使在100多个请求之后也没有增加.

Now, if I eliminate the "for" cycle, and only the query remains, the problem goes away, the memory usage remains at 190 Mb flat, no increase even after 100+ requests.

gc.collect()没有帮助.我还尝试查看函数开头和结尾处的tracemalloc统计信息的差异，但没有发现任何有用的信息.

gc.collect() at the end does not help. I have also tried looking at the difference in tracemalloc stats at the beginning and end of the function, I have not found anything useful.

请问有人经历过类似的事情吗?有什么想法可能会出问题吗?您可以推荐哪些其他测试/调查?这可能是我无法控制的Google App Engine/数据存储问题吗?

Has anyone experienced anything similar, please? Any ideas what might go wrong here? What additional tests / investigations can you recommend? Is this possibly a Google App Engine / Datastore issue I have no control of?

谢谢.

推荐答案

@Alex在另一个答案中进行了很好的研究，因此我将遵循以下建议:尝试使用

@Alex in the other answer did a pretty good research, so I will follow up with this recommendation: try using the NDB Library. All calls with this library have to be wrapped into a context manager, which should guarantee cleaning up after closing. That could help fix your problem:

ndb_client = ndb.Client(**init_client)

with ndb_client.context():
    query = MyModel.query().order(MyModel.my_column)
    sources = query.fetch()
    for source in sources:
        pass

# if you try to query DataStore outside the context manager, it will raise an error
query = MyModel.query().order(MyModel.my_column)

这篇关于Google App Engine/数据存储区/Flask/Python应用程序中的内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！