GAE Appstats RPC 时间线图显示复杂 NDB 查询的长时间延迟 [英] GAE Appstats RPC Timeline graph shows long delays with complex NDB queries

问题描述

我附上了来自下面我的生产应用搜索页面的 Appstats.该页面需要大约 45 秒才能通过 AJAX 加载结果.大约有 100 个实体.查询如下图:

I have attached the Appstats from my production app Search page below. The page takes ~45 seconds to load the results by AJAX. There are around 100 entities. The query is as shown below:

qry_1 = X.query(ndb.AND(X.active_status=="active", X.property_3==input_3, X.property_4==input_4, X.property_5==input_5, X.property_6.IN(input_6_list), X.property_20.IN(input_20_list))) 
record_list = qry_1.fetch() 

# input_6_list contains ~5 string items 
# input_20_list contains ~5 string items 

我无法弄清楚:
为什么下图中在 RPC 调用之间有空白显示.. 它们表示什么.. 我该如何防止它们，因为它们使我的网站无法使用
为什么查询需要这么长时间才能完成

I am not able to figure out:
why are there empty spaces shown in the graph below between the RPC calls.. what do they signify.. how can I prevent them as they make my website unusable
why is the query taking so long to complete

Call Trace 指向我代码中的以下行:

The Call Trace points to the following line in my code:

record_list = qry_1.fetch() 

显示它的调用跟踪线:

  <path[0]>/main.py:6332 post() 

推荐答案

自从 您之前的问题.您的数据看起来非常相关.该模型强制查询执行大量索引查找以满足 AND 和 IN 操作.简而言之，当前的模型永远不会扩展.它必须完全重组.

The situation has not really changed since your previous question. Your data looks very relational. The model is such that it forces queries to perform large numbers of index lookups to satisfy AND and IN operations. In short, the current model will never scale. It must be completely restructured.

从查询开始创建您的新结构 - 以最少的查询参数和索引查找输出所需数据的最快(换句话说，最简单)方法是什么?您可以连接所有旧的查询参数(属性)，对结果进行散列，并最终使用单个键在单个索引中进行查找.快速地.然后从那里向后工作以存储您的记录，以便属性的每个组合散列到唯一的键和相应的结果记录.您必须接受冗余(为不同的属性组合多次存储相同的结果)，但这是 NoSQL 方式，而不是第三范式.

Create your new structure by starting at the query - what is the fastest (in other words simplest) way to just output the required data with the minimum number of query parameters and index lookups? You could concatenate all your old query parameters (properties), hash the result, and end up with a single key to look up in a single index. Fast. Then work backwards from there to storing your records such that every combination of properties hashes to a unique key and a corresponding result record. You must accept redundancy (storing the same result multiple times for different property combinations) but that is the NoSQL way, not Third Normal Form.

在这些 StackOverflow 问题中，还有一些其他潜在有用的建议可以帮助您重新构建数据:存储歌曲、艺术家和专辑数据 和 如何存储文档结构.

There are some other potentially useful suggestions for re-structuring your data in these StackOverflow questions: Storing song, artist and album data and How to store document structure.

第二个答案:如果您的数据模型不灵活，请将 Datastore 替换为 Cloud SQL 因为在小型数据集上进行复杂查询时速度要快得多.

Second answer: If you are inflexible on your data model, replace Datastore with Cloud SQL because that is much faster with complex queries on small data sets.

这篇关于GAE Appstats RPC 时间线图显示复杂 NDB 查询的长时间延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！