Google App Engine数据库查询内存使用情况 [英] Google App Engine DB Query Memory Usage

问题描述

当我对一大组小对象（只有几个短的字符串和布尔属性的15k对象）运行查询时，没有对这些对象做任何事情，我看到我的实例的内存使用量不断增加（增加70Mb）。内存的增加看起来与它在内存中仅用于查询的数据量成正比。

我使用的循环如下所示：

如果光标为：b

  cursor = None 
 True：
 query = MyModel.all（） 
 query.run（batch_size = 500）：
 fetched = = 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 $在这里做一些'结果'。实际上为
＃测试保留空白，以确保我自己不保留任何东西
 
如果提取== 500：
 cursor = query.cursor（）
 break 
 else：
 break 
  

可以肯定这不是由于appstats造成的，我打电话给 appstats.recording.dont_record（）不记录任何统计信息。

有没有人有任何线索可能会继续？或任何关于如何调试/配置文件的指针？

更新1 ：我开启了 gc.set_debug（gc .DEBUG_STATS），并且我看到定期调用垃圾收集器，所以它 试图收集垃圾。当我在循环结尾（也是请求结束）调用 gc.collect（）时;它返回 0 ，并没有帮助。

更新2 ：我做了一些黑客让guppy在dev_appserver上工作，这似乎表明，在循环结束时显式地 gc.collect（）之后，大部分内存是由'google.appengine.datastore.entity_pb.Property'字典消耗。

解决方案

我已经将此报告给应用引擎团队，他们似乎确认这实际上是一个问题（怀疑是游标处理）。

When I run a query on a large set of small objects (15k objects with only a few short string and boolean properties), without doing anything with these objects, I see my instance's memory usage continuously increasing (70Mb increase). The memory increase doesn't look proportional to the amount of data it ever needs to keep in memory for just the query.

The loop I use is the following:

cursor = None
while True:
  query = MyModel.all()
  if cursor:
    query.with_cursor(cursor)
  fetched = 0
  for result in query.run(batch_size = 500):
    fetched += 1

    # Do something with 'result' here. Actually leaving it empty for 
    # testing to be sure I don't retain anything myself

    if fetched == 500:
      cursor = query.cursor()
      break
  else:
    break

To be sure this is not due to appstats, I call appstats.recording.dont_record() to not record any stats.

Does anyone have any clue what might be going on? Or any pointers on how to debug/profile this?

Update 1: I turned on gc.set_debug(gc.DEBUG_STATS) on the production code, and I see the garbage collector being called regularly, so it is trying to collect garbage. When I call a gc.collect() at the end of the loop (also the end of the request); it returns 0, and doesn't help.

Update 2: I did some hacking to get guppy to work on dev_appserver, and this seemed to point that, after an explicit gc.collect() at the end of the loop, most of the memory was consumed by a 'dict of google.appengine.datastore.entity_pb.Property'.

解决方案

I have reported this to the app engine team, and they seem to confirm this is actually a problem (suspected to be with the handling of cursors).

这篇关于Google App Engine数据库查询内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！